system

Abstract

We provide the system. [Solution] A means for activating a voice response system when a communication device receives an incoming call and obtaining voice data from the caller, A means of converting audio data into text data and summarizing it using natural language processing, A means of comparing the sender's identification information with a database and calculating the reliability, A means of making decisions to allow or deny a call based on trustworthiness, A means of analyzing call content and issuing a warning if an anomaly is detected, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In modern society where fraud targeting the elderly and dementia patients and nuisance sales calls routinely received by companies and individuals are increasing, it is becoming difficult for recipients to quickly and safely receive only legitimate calls. Such a situation poses a problem with significant risks especially for users with reduced judgment. Therefore, there is a need for effective means to pre-filter illegal and nuisance calls and appropriately receive only reliable calls.

Means for Solving the Problems

[0005] This invention introduces a means for activating an AI-based voice response system when a communication device receives an incoming call, converting the caller's voice data into text data, and summarizing it using natural language processing. It also provides a method for calculating a reliability score by comparing the caller's identification information with a database and deciding whether to allow or deny the call based on that reliability score. Furthermore, it establishes a mechanism to protect users from fraudulent calls by analyzing the call content and issuing a warning if any abnormalities are detected. This configuration makes it possible to safely provide only legitimate calls while eliminating unnecessary calls.

[0006] A "communication device" is an electronic device used to send and receive voice and data via telephone lines or networks.

[0007] A "voice response system" is a program that automatically responds to callers with voice messages upon receiving an incoming call and collects information from them.

[0008] "Caller" refers to a person or organization that initiates a call using a communication device.

[0009] "Audio data" refers to information that represents sound in digital format, and is a component of an audio signal.

[0010] "Text data" refers to digital data that represents audio as written text.

[0011] "Natural language processing" is a technology that enables computers to understand human language and analyze and generate meaning.

[0012] "Summarization" is the process of extracting only the main points of information while preserving the overall picture, and expressing them in a shortened form.

[0013] "Identification information" refers to information used to identify the sender and verify their identity and characteristics.

[0014] A "database" is a system constructed to efficiently store, search, and reference specific information.

[0015] "Reliability" is an index that quantifies or evaluates the degree to which certain information or a person can be trusted.

[0016] "Analysis" is an act of decomposing or observing a target in detail to investigate it and understand its characteristics and structure.

[0017] "Warning" is a notice or message issued to prompt attention.

Brief Explanation of Drawings

[0018] [Figure 1] It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing apparatus and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing apparatus and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing apparatus and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing apparatus and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11]It is a sequence diagram showing the processing flow of the data processing system in Embodiment 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Embodiment 2 when combined with an emotion engine. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when combined with an emotion engine.

Mode for Carrying Out the Invention

[0019] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0022] In the following embodiments, a numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0023] In the following embodiments, the signed storage is one or more non-volatile storage devices that store various programs and various parameters. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes.

[0024] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0026] [First Embodiment]

[0027] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0028] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0031] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0034] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0038] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0039] This invention provides a system that enables users, including the elderly and those with dementia, to engage in safe and reliable communication. Specific embodiments of the system are described below.

[0040] In this invention, the server acts as a communication device, receiving all incoming calls first. Upon receiving an incoming call, the server activates an AI-powered voice response system and automatically sends a message to the caller saying, "Please tell us your name and purpose of your call." The caller's voice response is converted into text data in real time by the terminal.

[0041] The terminal summarizes this text data using natural language processing technology and stores it in a structured format. Next, the server matches the caller's identification information (e.g., phone number) against a dedicated database. This database stores information on past communication history and the trustworthiness of each caller, and a trust score for the caller is calculated based on this information.

[0042] The confidence score serves as a criterion for the server to decide whether to allow a call. If the confidence score exceeds a certain level, the server will make the user's device ring to inform the user of the intention to call. This allows the user to receive only calls from trusted callers. On the other hand, if the confidence score is low, the server will automatically terminate the call or switch to voicemail mode.

[0043] Even after the user answers a call, the device continues to monitor the call content in real time. If any unusual patterns or content that does not match the initial summary is detected during the call, the device can alert the user. For example, if an inappropriate request suddenly appears, the device will automatically sound an alert. This feature allows users to make calls with greater peace of mind.

[0044] A concrete example is a case where a server receives an incoming call, and the caller, claiming to be a "financial institution representative," attempts to verify account information. The server checks the caller's number in its database and calculates its reliability by referring to past records. If the number matches one suspected of being a fraudulent call, the server rejects the call and does not notify the user. In this way, the present invention prevents fraudulent communications and protects users securely.

[0045] The following describes the processing flow.

[0046] Step 1:

[0047] When the server detects an incoming call, it activates the voice response system. It automatically sends a message to the caller saying, "Please tell me your name and how can I help you?"

[0048] Step 2:

[0049] When the caller responds to a message, the device converts the audio data into text data in real time. Speech recognition technology is used to convert the caller's speech into digital text information.

[0050] Step 3:

[0051] The text data acquired by the terminal is fed into a natural language processing engine for summarization. This generates a concise summary of the caller's name and the purpose of the message.

[0052] Step 4:

[0053] The server checks the caller's phone number against a database and calculates a confidence score based on past communication history and ratings. It utilizes information from reliable data sources to quantify the caller's trustworthiness.

[0054] Step 5:

[0055] The server evaluates the trust score and decides whether to ring the user's device. If the trust score exceeds the threshold, the device rings to allow the user to make a call. If the trust score is low, the call is automatically rejected or switched to voicemail mode.

[0056] Step 6:

[0057] When a user starts a call, the device monitors the call content in real time. It transcribes the audio data during the call and verifies whether it matches the initial summary.

[0058] Step 7:

[0059] If unusual content or inconsistencies with the initial summary are detected during a call, the device will alert the user. If necessary, it will automatically terminate the call to protect the user.

[0060] By following the steps outlined above, this system can provide users with a safe and reliable calling environment.

[0061] (Example 1)

[0062] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0063] The goal is to provide a system that allows users, including the elderly and those with dementia, to communicate safely and securely when they receive fraudulent calls or unauthorized communications. Conventional communication methods make it difficult to accurately determine the caller's intentions, necessitating appropriate security measures. This system aims to prevent confusion and problems caused by incoming calls from unreliable callers.

[0064] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0065] In this invention, the server includes means for activating a voice response function using a generative model when a communication device receives an incoming call, presenting a prompt message to the caller to obtain voice information, converting the voice information into text and summarizing it using a natural language processing function, and comparing the caller's identification number with an information base to calculate a reliability rating. As a result, users can receive only communications from trusted callers, eliminate abnormal communications in advance, and maintain a safe and secure communication environment.

[0066] A "communication device" is a device that can send and receive voice and data, and has the function of starting the next processing step upon receiving an incoming call.

[0067] A "generative model" refers to an algorithm that operates using artificial intelligence, understands human language, and generates responses, and is used to drive voice response functions.

[0068] A "prompt message" is a message designed to prompt the caller for information or a response, and refers to questions or instructions automatically generated by voice response functions.

[0069] "Audio information" refers to audio data emitted by a caller, which is converted into text through speech recognition processing.

[0070] "Natural language processing" refers to the technology that allows computers to understand, interpret, and generate human language, and is used to summarize text information.

[0071] An "identification value" is numerical data used to identify the sender and verify the origin of the message.

[0072] An "information base" refers to a data storage system that accumulates past communication history and sender reliability information, and is used as a criterion for reliability evaluation.

[0073] "Trust rating" is a numerical representation of the trustworthiness of a sender, calculated based on their past history and evaluations.

[0074] "Call information" refers to the audio and text data generated during a call, and is the content of the communication that is subject to analysis.

[0075] This invention provides a system that enables users, including the elderly and those with dementia, to communicate safely and reliably. The system is comprised of a combination of voice response functionality, speech recognition technology, natural language processing technology, identification information matching, reliability evaluation, and call monitoring.

[0076] The server operates as a communication device and receives incoming calls from callers. Upon receiving a call, the server uses a generative AI model to activate its voice response function. This function generates a prompt message and presents it to the caller. An example of a prompt message used is, "Please tell us your name and purpose of your call." This prompt message serves to encourage the caller to provide a clear response.

[0077] The device utilizes speech recognition technology to convert the caller's voice information into text in real time. Specifically, general-purpose speech recognition software is used. The converted text data is summarized using natural language processing techniques. This allows the caller's requirements and intentions to be clarified in a short amount of time.

[0078] The server compares the caller's identification number with an information base and calculates a trust rating. This information base stores past communication history and ratings of the caller's trustworthiness. Based on this trust rating, the server decides whether to allow or deny the call.

[0079] Even if the user answers, the terminal continues to analyze the call information, and if it detects suspicious patterns or content that does not match the initial summary, it will warn the user. In this way, users can communicate securely and reliably.

[0080] A concrete example is a case where the caller impersonates a financial institution employee and attempts to verify account information. In this case, the server checks the caller's identification number and rejects the call if it is suspected to be fraudulent. At the same time, the user receives a warning about the suspicious content. This system prevents fraudulent communication and ensures the safety of users.

[0081] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0082] Step 1:

[0083] The server receives the incoming call. The input is an audio signal along with the caller's identifier. This incoming signal invokes the voice response function. The output is a preparation to issue a prompt to the caller using a generative AI model. This prompt is "Please tell us your name and purpose of call."

[0084] Step 2:

[0085] The device converts the caller's voice response into text data in real time. It receives the voice signal as input and digitizes it as text data using speech recognition technology. The output is text data in string format. Specifically, the content spoken by the caller is accurately transcribed into text.

[0086] Step 3:

[0087] The terminal summarizes this text data using natural language processing techniques. The input is the text data obtained in step 2. A natural language processing algorithm is applied to this data to extract important information and generate a simplified summary. The output is a summarized text, making the sender's requirements easier to understand.

[0088] Step 4:

[0089] The server compares the caller's identification number against a dedicated information base. The input is the caller identification data received in step 1. By comparing it with past communication history stored in the information base, the server calculates a trust rating for the caller. The output is a trust score. Based on this score, the server is ready to decide whether to allow or deny the call.

[0090] Step 5:

[0091] The server decides how to handle the call based on the trust score. The input is the trust score calculated in step 4. If the trust score is above a certain level, the server rings the user's device and allows the call. Conversely, if the trust score is low, the server rejects the call or switches to voicemail mode. The output is whether the call was allowed or terminated.

[0092] Step 6:

[0093] When a user receives a call, the device monitors the call content. It converts the audio data into text in real time and verifies that it matches the initial summary. The input is the audio data generated during the call. As output, it warns the user if any unusual patterns are detected. Specifically, an alert will sound if a sudden, inappropriate request occurs.

[0094] (Application Example 1)

[0095] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0096] The communication risks that users, including the elderly and those with dementia, face on a daily basis can cause serious problems, particularly through fraudulent attempts such as scams. This invention aims to prevent such fraudulent communications and provide a safe and reliable calling experience. In particular, it is necessary to provide a method for detecting fraudulent calls in real time and preventing harm to users.

[0097] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0098] In this invention, the server includes means for activating a voice response system when a communication device receives an incoming call and acquiring voice data from the caller; means for converting the voice data into text data and summarizing it using natural language processing; means for comparing the caller's identification information with a database and calculating the reliability level; and means for re-evaluating the reliability level in real time and allowing only secure calls. This makes it possible to detect fraudulent communications as suspicious patterns and issue warnings to users.

[0099] "Communication equipment" refers to devices and means for receiving incoming calls, and plays a central role in acquiring and processing voice data.

[0100] A "voice response system" is a program or function that automatically provides a voice message to the caller upon receiving a call and collects necessary information.

[0101] "Audio data" refers to audio information obtained from the caller, and is subject to analysis and text conversion.

[0102] "Text data" refers to the written information that remains after audio data has been converted, and is a data format used for natural language processing and summarization.

[0103] "Natural language processing" refers to the technology of analyzing and processing human language using computers, and is a technical means of summarizing text data.

[0104] "Caller identification information" refers to data used to identify a caller, such as a phone number or other unique information.

[0105] A "database" is a collection of data that stores past communication history and evaluations, and is used as reference information for calculating reliability.

[0106] "Trustworthiness" is an indicator used to evaluate the sender's communication behavior and demonstrate the security and legitimacy of the communication.

[0107] "Real-time" is a term that indicates that information acquisition and processing occur virtually instantaneously.

[0108] "Anomaly" refers to behavior that deviates from normal communication patterns or suspicious points that suggest a risk of fraud.

[0109] This invention relates to a system for enabling safe and reliable communication for users, including the elderly and those with dementia. A server functions as the central component of this system and is implemented as follows:

[0110] When the server receives an incoming call, it automatically provides a message to the caller using a voice response system and collects the necessary voice data. This voice data is converted into text data in real time on the terminal. Software such as TENSORFLOW® or natural language processing libraries (e.g., spaCy) are used for the voice-to-text conversion.

[0111] Furthermore, the server matches the caller's identification information against a database and refers to past communication history in the database to calculate the reliability score. This reliability score is updated in real time, enhancing the security of calls.

[0112] The user's device has a feature that only notifies the user of incoming calls if the call's trustworthiness exceeds a certain threshold. This ensures that the user only receives calls from trusted sources. If a suspicious pattern is detected, the device will alert the user. This strengthens protection, especially against potentially fraudulent calls.

[0113] For example, if a user receives a call claiming to be from the city hall with an "important announcement," the server will automatically block the call if the caller's trustworthiness is low. This also helps the user prevent becoming a victim of fraud. To standardize this response process, the following is used as an example of a prompt:

[0114] "Input: Convert the recorded audio of a phone call from the city hall into text and analyze it for any suspicious patterns."

[0115] "Output: Confidence score and warning message for suspicious patterns"

[0116] Thus, the system according to the present invention aims to provide a safe and smooth communication experience for the elderly and dementia patients.

[0117] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0118] Step 1:

[0119] When the server receives an incoming call, it activates the voice response system and automatically sends a message to the caller. The input is an incoming call, and the output is a voice message to the caller. This process captures the caller's voice data.

[0120] Step 2:

[0121] The device converts acquired audio data into text data in real time. The input is the acquired audio data, and the output is the corresponding text data. Speech recognition technology is used here; for example, TensorFlow is used to convert audio to text.

[0122] Step 3:

[0123] The server applies natural language processing to text data to create a summary. The input is text data, and the output is summarized text. Here, a generative AI model is used to summarize the data, structuring the information.

[0124] Step 4:

[0125] The server compares the caller's identification information with a database and calculates the confidence score. The input is the caller's identification information and the database used for comparison, and the output is the confidence score. This confidence score is used to evaluate the caller's safety.

[0126] Step 5:

[0127] The server decides whether to allow or deny a call based on a confidence score. The input is the confidence score, and the output is whether the call is allowed or denied. Based on the set threshold, the call is only allowed if the caller is deemed trustworthy.

[0128] Step 6:

[0129] If a call is permitted, the user's device analyzes the call content in real time and monitors for any abnormalities. The input is the call content, and the output is whether or not there are any abnormalities. This monitoring allows the user to confirm that they are safe even while on a call.

[0130] Step 7:

[0131] If an anomaly is detected, the device will immediately issue a warning to the user. The input is the analyzed call content, and the output is a warning notification. Specifically, if a suspicious request is detected, the device will alert the user by displaying a warning sound or notification.

[0132] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0133] This invention is a system that combines an emotion engine for recognizing user emotions with an incoming call management system in a communication device. Specific embodiments of this system are described below.

[0134] When the server detects an incoming call, it activates the voice response system and requests a voice response from the caller. The caller's response is received as voice data by the terminal and converted to text. This text data is summarized using natural language processing techniques, and the caller's name and request are analyzed.

[0135] Simultaneously, the server compares the caller's identification information with a database and calculates a confidence score based on past communication history and evaluations. Based on this confidence score, the server decides whether to allow the user to make a call.

[0136] Furthermore, during a call, the device's built-in emotion engine analyzes the user's voice tone, speaking speed, and word choice to recognize emotions in real time. The emotion engine detects basic emotions such as joy, anger, surprise, sadness, and anxiety, and supports decision-making based on the communication situation. Based on this information, the server can decide whether to continue the call, interrupt it, or issue additional warnings.

[0137] As a concrete example, consider a scenario where a user answers a call and the caller begins speaking in an aggressive tone. The emotion engine recognizes this as "anger," and if the user starts to feel stressed, the device can emit a warning sound or display a message on the screen asking, "Do you want to continue the call?" There is also an option to automatically terminate the call if the emotional escalation occurs.

[0138] Thus, by combining emotion analysis using an emotion engine and realizing even more advanced call management, the present invention can provide users with a safer and more comfortable calling environment.

[0139] The following describes the processing flow.

[0140] Step 1:

[0141] When the server detects an incoming call, it activates the voice response system. The server automatically sends a voice message to the caller saying, "Please tell me your name and how can I help you?"

[0142] Step 2:

[0143] When the caller answers, the terminal acquires the audio data and converts it into text data in real time. Speech recognition technology is used in this process.

[0144] Step 3:

[0145] The terminal feeds the text data into a natural language processing engine to generate a summary. This summarizes the caller's name and the main points of the request.

[0146] Step 4:

[0147] The server checks the caller's phone number against a database. The database contains records of past communication history and ratings, and the server calculates a confidence score based on this information.

[0148] Step 5:

[0149] The server evaluates the confidence score and decides whether to ring the user's device. If the score exceeds a certain threshold, the device will ring to inform the user of the purpose of the call. Otherwise, the call will be automatically rejected or switched to voicemail mode.

[0150] Step 6:

[0151] When a user initiates a call, the device activates its emotion engine. The emotion engine analyzes the user's voice tone and speaking speed in real time and evaluates their emotional state.

[0152] Step 7:

[0153] When the emotion engine detects stressful states such as "anger" or "anxiety," the device issues a warning to the user. Specifically, it sounds a warning and displays a warning message on the screen.

[0154] Step 8:

[0155] In response to the device's warning, the user can choose to continue or end the call. The system can also automatically terminate the call if there are signs of emotional escalation.

[0156] Through this series of processing steps, the system provides users with a safe and comfortable calling environment.

[0157] (Example 2)

[0158] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0159] Modern communication devices are required to appropriately assess caller information and emotional state during calls to provide a safe and comfortable calling experience. However, conventional systems have not adequately verified caller information or analyzed emotions during calls, potentially leading to unpleasant experiences for users. Therefore, the challenge is to provide technology that enables appropriate call management based on caller reliability and emotional changes during calls.

[0160] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0161] In this invention, the server includes means for activating a voice response function when a communication device detects an incoming call and obtaining voice information from the caller; means for converting the voice information into text information and summarizing it using information processing technology; means for comparing the caller's identification information with a recording device and calculating the reliability level; means for analyzing emotions in real time during a call using an information analysis engine and providing warnings according to those emotions; and means for automatically terminating a call when emotions exceed a certain threshold. This enables users to have a safe and comfortable calling environment.

[0162] "Communication equipment" refers to devices used to send and receive voice and data, and includes telephones, smartphones, and network devices.

[0163] "Voice response function" refers to a system in which a communication device automatically receives voice messages and generates the necessary response.

[0164] "Audio information" refers to data transmitted through human voices, and includes phone calls and recorded audio.

[0165] "Text information" refers to information expressed as a string of characters, and includes data obtained by converting audio information into text.

[0166] "Information processing technology" refers to computer technologies used to analyze, transform, and utilize data, and includes natural language processing, among others.

[0167] "Identification information" refers to data used to identify a specific individual or sender, and includes names and ID numbers.

[0168] A "recording device" refers to a device for storing information, and this includes databases and storage systems.

[0169] "Trustworthiness" is a value that indicates how trustworthy a particular piece of information or its source is, and is calculated based on past history and evaluations.

[0170] An "information analysis engine" refers to software or algorithms used to analyze data and extract meaning and patterns.

[0171] "Emotions" refer to the psychological state exhibited by the caller and receiver during a phone call, and include feelings such as joy, anger, surprise, sadness, and anxiety.

[0172] A "standard" refers to a benchmark or reference point used when making a particular action or judgment.

[0173] This invention combines an emotion recognition engine with an incoming call management system for communication devices, providing a system that enables users to have safer and more comfortable phone calls. This system mainly consists of a server and terminals, each performing its own specific role.

[0174] The server's processing begins when the communication device detects an incoming call; it activates the voice response function to obtain voice information from the caller. The voice response system uses any speech synthesis technology to request the caller to provide their name and purpose of the call. The acquired voice information is then converted into text on the server. For speech recognition, commonly used technologies such as Google® Speech-to-Text API or open-source speech recognition libraries can be utilized.

[0175] The server analyzes the converted text information using information processing technology and calculates a confidence score by comparing the caller's identification information with that of a recording device. This can be done by applying natural language processing technology and by incorporating libraries such as Apache® OpenNLP and Stanford NLP. The confidence score thus calculated is used as a criterion for deciding whether to allow or deny the call.

[0176] During a call, the device uses an information analysis engine to analyze the user's emotions in real time. This emotion engine employs technologies such as Affectiva and Emotion AI to identify emotions from changes in the user's voice tone and speaking style. This emotion data is sent to a server and contributes to issuing warnings based on the communication status and determining when to end the call.

[0177] As a concrete example, consider a scenario where the caller's voice becomes aggressive during a call. When the emotion engine recognizes "anger," the device emits a warning sound and displays the option "Do you wish to continue the call?" on the screen. Furthermore, a feature is provided that allows the user to automatically terminate the call if such emotions escalate.

[0178] A concrete example of a prompt for a generative AI model could be: "What is the most effective way to warn a user when they are receiving a call from someone speaking in an aggressive tone?"

[0179] This system allows users to enjoy a more comfortable and secure calling environment based on the caller's trustworthiness and emotional state during the call.

[0180] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0181] Step 1:

[0182] The server detects an incoming call from the communication device and activates the voice response function. The server receives the incoming signal as input and issues a command to activate the voice response system. This process prepares the server to begin providing a voice message to the caller.

[0183] Step 2:

[0184] The server obtains voice information from the caller. The voice response system receives the caller's response in voice format and sends it to the server. The input is the caller's raw voice data, which is stored on the server for later processing.

[0185] Step 3:

[0186] The server converts the audio information into text information. The server uses speech recognition software to process the acquired audio data and generate text data. This conversion makes the audio content available as text information.

[0187] Step 4:

[0188] The server analyzes and summarizes text information. Utilizing natural language processing technology, it processes the input text information to extract important elements such as the caller's name and purpose. The output is summarized information, which can be used for call management.

[0189] Step 5:

[0190] The server compares the caller's identification information with the recording device and calculates the confidence level. The server takes caller identification data as input and evaluates the confidence level by comparing it with a database. The confidence level calculated based on past communication history is stored as output within the server.

[0191] Step 6:

[0192] The server decides whether to allow or deny the call based on its trustworthiness. Based on the acquired trustworthiness, the server makes a decision: if it allows the call, it continues the call; if it denies it, it terminates the call. The output is determined as either the allowed or denied status of the call.

[0193] Step 7:

[0194] The device analyzes the user's emotions in real time during a call. The device uses a data analysis engine to identify emotions based on the user's recorded voice. The recognized emotion information is sent to a server and used to inform communication decisions.

[0195] Step 8:

[0196] The server provides users with emotionally-sensitive warnings. Based on emotional information, the server displays warning sounds or messages to the user via the terminal if necessary, ensuring the safety of the call. It receives emotional information as input and utilizes warning methods as output.

[0197] Step 9:

[0198] The server automatically terminates the call when emotions exceed a certain threshold. The server compares emotional information with the set threshold and, if the threshold is exceeded, automatically outputs a command to end the call. This action helps users avoid unnecessary stress and unpleasant situations.

[0199] (Application Example 2)

[0200] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0201] There is a need to improve the safety and comfort of calls using communication devices. In particular, there is a lack of means to recognize the caller's emotions and stress levels in real time, which presents a challenge in responding appropriately to situations where users feel anxious or stressed. In this situation, an effective system is needed to mitigate the risk of users encountering unforeseen circumstances.

[0202] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0203] In this invention, the server includes means for activating a voice response system when a communication device receives an incoming call and acquiring voice data from the caller, means for analyzing voice tone and speaking speed to recognize emotions in real time, and means for issuing an alert when the user feels stressed. This makes it possible to analyze the caller's emotional state and provide appropriate feedback to the user.

[0204] A "communication device" is a device that transmits and receives voice data, enabling communication between users.

[0205] A "voice response system" is a system that automatically activates when a call is received and receives voice data from the caller.

[0206] "Audio data" refers to data that records the speaker's utterance as a digital signal.

[0207] "Natural language processing" is an information processing technology that analyzes text data and derives its meaning and summary.

[0208] "Caller identification information" refers to information used to identify a caller, and includes information such as a phone number and name.

[0209] "Trustworthiness" is an indicator of whether a sender is trustworthy, and it is a score based on past communication history and evaluations.

[0210] "Voice tone" refers to the characteristics of pitch and intensity of a voice that reflect the speaker's psychological state and emotions.

[0211] "Speech rate" is an indicator that shows the number of words a speaker utters within a certain period of time, and is a standard for measuring the pace of communication.

[0212] "Means of recognizing emotions in real time" refers to technologies and methods for instantly determining the emotional state of a speaker while they are speaking.

[0213] "Means of issuing alerts" refers to notification functions used to warn or alert users.

[0214] To realize this invention, it is necessary to install a dedicated application on communication devices such as smartphones and servers. The main functions of the system are voice data processing and emotion recognition, and it operates as follows.

[0215] First, when a communication device receives an incoming call, the server activates the voice response system and retrieves the caller's voice data. This data is converted into text data using a speech recognition API (e.g., Google Cloud Speech-to-Text). Next, the text is summarized using a natural language processing library (e.g., spaCy), and the caller's identification information and confidence level are calculated through database matching.

[0216] The server further analyzes the received voice tone and speaking speed in real time using an emotion recognition engine built into the device. This analysis detects the caller's emotions, such as joy or anger. If the user feels stressed, the device issues an alert and presents the user with options such as "Do you want to continue the call?". It can also automatically terminate the call if the emotions escalate.

[0217] For example, if the system detects unsettling emotions during a call, it will automatically trigger a prompt in real time to "monitor the caller's emotions and notify the user if their tone is unsettling." This mechanism protects users from unexpected and unpleasant interactions.

[0218] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0219] Step 1:

[0220] The server activates the voice response system when the communication device receives an incoming call and acquires voice data from the caller. The input at this time is a real-time voice signal, and the output is the acquired voice data. The voice response system uses a microphone to capture the voice signal.

[0221] Step 2:

[0222] The server converts the acquired audio data into text data using a speech recognition API. The input is audio data, and the output is the corresponding text data. A speech recognition service such as Google Cloud Speech-to-Text is used for this conversion.

[0223] Step 3:

[0224] The server analyzes text data using a natural language processing library and summarizes the sender's requirements. The input is text data, and the output is the summarized content. A natural language processing tool like spaCy extracts keywords and context through the analysis.

[0225] Step 4:

[0226] The server compares the caller's identification information with a database and calculates the confidence score. The input is the caller's identification information, and the output is the caller's confidence score. The calculation is based on past communication history and evaluations.

[0227] Step 5:

[0228] The server decides whether to allow or deny a call based on a confidence score. The input is the confidence score, and the output is whether the call is allowed or denied. If the confidence score falls below a certain threshold, the call may be denied.

[0229] Step 6:

[0230] The server uses the emotion recognition engine built into the terminal to analyze the voice tone and speech rate during communication in real time. The input is text data and voice characteristics, and the output is the emotion analysis result. A generative AI model detects emotions and determines specific emotional states such as "joy" or "anger."

[0231] Step 7:

[0232] The device issues an alert when the user experiences stress. The input is the result of sentiment analysis, and the output is an alert to the user. The alert is delivered to the user as a visual or audio notification.

[0233] Step 8:

[0234] An interface is provided for the user to choose a course of action. The input is the user's selection, and the output is an action such as continuing or ending the call. A prompt such as "Do you want to continue the call?" is displayed to the user.

[0235] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0236] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0237] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0238] [Second Embodiment]

[0239] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0240] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0241] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0242] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0243] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0244] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0245] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0246] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0247] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0248] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0249] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0250] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0251] This invention provides a system that enables users, including the elderly and those with dementia, to engage in safe and reliable communication. Specific embodiments of the system are described below.

[0252] In this invention, the server acts as a communication device, receiving all incoming calls first. Upon receiving an incoming call, the server activates an AI-powered voice response system and automatically sends a message to the caller saying, "Please tell us your name and purpose of your call." The caller's voice response is converted into text data in real time by the terminal.

[0253] The terminal summarizes this text data using natural language processing technology and stores it in a structured format. Next, the server matches the caller's identification information (e.g., phone number) against a dedicated database. This database stores information on past communication history and the trustworthiness of each caller, and a trust score for the caller is calculated based on this information.

[0254] The confidence score serves as a criterion for the server to decide whether to allow a call. If the confidence score exceeds a certain level, the server will make the user's device ring to inform the user of the intention to call. This allows the user to receive only calls from trusted callers. On the other hand, if the confidence score is low, the server will automatically terminate the call or switch to voicemail mode.

[0255] Even after the user answers a call, the device continues to monitor the call content in real time. If any unusual patterns or content that does not match the initial summary is detected during the call, the device can alert the user. For example, if an inappropriate request suddenly appears, the device will automatically sound an alert. This feature allows users to make calls with greater peace of mind.

[0256] A concrete example is a case where a server receives an incoming call, and the caller, claiming to be a "financial institution representative," attempts to verify account information. The server checks the caller's number in its database and calculates its reliability by referring to past records. If the number matches one suspected of being a fraudulent call, the server rejects the call and does not notify the user. In this way, the present invention prevents fraudulent communications and protects users securely.

[0257] The following describes the processing flow.

[0258] Step 1:

[0259] When the server detects an incoming call, it activates the voice response system. It automatically sends a message to the caller saying, "Please tell me your name and how can I help you?"

[0260] Step 2:

[0261] When the caller responds to a message, the device converts the audio data into text data in real time. Speech recognition technology is used to convert the caller's speech into digital text information.

[0262] Step 3:

[0263] The text data acquired by the terminal is fed into a natural language processing engine for summarization. This generates a concise summary of the caller's name and the purpose of the message.

[0264] Step 4:

[0265] The server checks the caller's phone number against a database and calculates a confidence score based on past communication history and ratings. It utilizes information from reliable data sources to quantify the caller's trustworthiness.

[0266] Step 5:

[0267] The server evaluates the trust score and decides whether to ring the user's device. If the trust score exceeds the threshold, the device rings to allow the user to make a call. If the trust score is low, the call is automatically rejected or switched to voicemail mode.

[0268] Step 6:

[0269] When a user starts a call, the device monitors the call content in real time. It transcribes the audio data during the call and verifies whether it matches the initial summary.

[0270] Step 7:

[0271] If unusual content or inconsistencies with the initial summary are detected during a call, the device will alert the user. If necessary, it will automatically terminate the call to protect the user.

[0272] By following the steps outlined above, this system can provide users with a safe and reliable calling environment.

[0273] (Example 1)

[0274] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0275] The goal is to provide a system that allows users, including the elderly and those with dementia, to communicate safely and securely when they receive fraudulent calls or unauthorized communications. Conventional communication methods make it difficult to accurately determine the caller's intentions, necessitating appropriate security measures. This system aims to prevent confusion and problems caused by incoming calls from unreliable callers.

[0276] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0277] In this invention, the server includes means for activating a voice response function using a generative model when a communication device receives an incoming call, presenting a prompt message to the caller to obtain voice information, converting the voice information into text and summarizing it using a natural language processing function, and comparing the caller's identification number with an information base to calculate a reliability rating. As a result, users can receive only communications from trusted callers, eliminate abnormal communications in advance, and maintain a safe and secure communication environment.

[0278] A "communication device" is a device that can send and receive voice and data, and has the function of starting the next processing step upon receiving an incoming call.

[0279] A "generative model" refers to an algorithm that operates using artificial intelligence, understands human language, and generates responses, and is used to drive voice response functions.

[0280] A "prompt sentence" is a message for prompting information or responses to a sender, and refers to inquiries or guidance automatically generated by a voice response function.

[0281] "Voice information" refers to voice data emitted by a sender, which is the target to be converted into text by voice recognition processing.

[0282] The "natural language processing function" refers to the technology of enabling a computer to understand, interpret, and generate human language, and is used for summarizing text information.

[0283] An "identification number" is numerical data for identifying a sender, and is used for verifying the source of transmission.

[0284] An "information base" refers to a data storage that accumulates past communication histories and sender trust information, and is used as a criterion for trust evaluation.

[0285] "Trust evaluation" is a quantification of the reliability of a sender, and is an index calculated based on past histories and evaluations.

[0286] "Call information" refers to voice data and text data generated during a call, and refers to the communication content to be analyzed.

[0287] This invention provides a system that enables users including the elderly and dementia patients to conduct safe and reliable communications. This system is configured by combining a voice response function, voice recognition technology, natural language processing technology, verification of identification information, trust evaluation, and call monitoring.

[0288] The server operates as a communication device and receives incoming calls from callers. Upon receiving a call, the server uses a generative AI model to activate its voice response function. This function generates a prompt message and presents it to the caller. An example of a prompt message used is, "Please tell us your name and purpose of your call." This prompt message serves to encourage the caller to provide a clear response.

[0289] The device utilizes speech recognition technology to convert the caller's voice information into text in real time. Specifically, general-purpose speech recognition software is used. The converted text data is summarized using natural language processing techniques. This allows the caller's requirements and intentions to be clarified in a short amount of time.

[0290] The server compares the caller's identification number with an information base and calculates a trust rating. This information base stores past communication history and ratings of the caller's trustworthiness. Based on this trust rating, the server decides whether to allow or deny the call.

[0291] Even if the user answers, the terminal continues to analyze the call information, and if it detects suspicious patterns or content that does not match the initial summary, it will warn the user. In this way, users can communicate securely and reliably.

[0292] A concrete example is a case where the caller impersonates a financial institution employee and attempts to verify account information. In this case, the server checks the caller's identification number and rejects the call if it is suspected to be fraudulent. At the same time, the user receives a warning about the suspicious content. This system prevents fraudulent communication and ensures the safety of users.

[0293] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0294] Step 1:

[0295] The server receives the incoming call. The input is an audio signal along with the caller's identifier. This incoming signal invokes the voice response function. The output is a preparation to issue a prompt to the caller using a generative AI model. This prompt is "Please tell us your name and purpose of call."

[0296] Step 2:

[0297] The device converts the caller's voice response into text data in real time. It receives the voice signal as input and digitizes it as text data using speech recognition technology. The output is text data in string format. Specifically, the content spoken by the caller is accurately transcribed into text.

[0298] Step 3:

[0299] The terminal summarizes this text data using natural language processing techniques. The input is the text data obtained in step 2. A natural language processing algorithm is applied to this data to extract important information and generate a simplified summary. The output is a summarized text, making the sender's requirements easier to understand.

[0300] Step 4:

[0301] The server compares the caller's identification number against a dedicated information base. The input is the caller identification data received in step 1. By comparing it with past communication history stored in the information base, the server calculates a trust rating for the caller. The output is a trust score. Based on this score, the server is ready to decide whether to allow or deny the call.

[0302] Step 5:

[0303] The server determines the call processing according to the trust score. The input is the trust score calculated in step 4. If the trust score is above a certain level, the user's terminal is rung to permit the call. Conversely, if the trust score is low, the call is rejected or switched to the voicemail mode. The output is the state of call permission or termination.

[0304] Step 6:

[0305] When the user receives a call, the terminal monitors the call content. The voice data is converted into text in real time and checked whether it matches the initial summary content. The input is the voice data generated during the call. As an output, when an abnormal pattern is detected, a warning is issued to the user. Specifically, an alert sounds when a sudden improper request or the like occurs.

[0306] (Application Example 1)

[0307] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as the "server", and the smart glasses 214 are referred to as the "terminal".

[0308] The communication risks that users including the elderly and dementia patients face daily can cause serious problems especially by fraudulent attempts such as fraud. The present invention aims to prevent such illegal communications and provide a safe and reliable call experience. In particular, it is necessary to provide a method for detecting fraudulent calls in real time and preventing damage to users.

[0309] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0310] In this invention, the server includes means for activating a voice response system when a communication device receives an incoming call and acquiring voice data from the caller; means for converting the voice data into text data and summarizing it using natural language processing; means for comparing the caller's identification information with a database and calculating the reliability level; and means for re-evaluating the reliability level in real time and allowing only secure calls. This makes it possible to detect fraudulent communications as suspicious patterns and issue warnings to users.

[0311] "Communication equipment" refers to devices and means for receiving incoming calls, and plays a central role in acquiring and processing voice data.

[0312] A "voice response system" is a program or function that automatically provides a voice message to the caller upon receiving a call and collects necessary information.

[0313] "Audio data" refers to audio information obtained from the caller, and is subject to analysis and text conversion.

[0314] "Text data" refers to the written information that remains after audio data has been converted, and is a data format used for natural language processing and summarization.

[0315] "Natural language processing" refers to the technology of analyzing and processing human language using computers, and is a technical means of summarizing text data.

[0316] "Caller identification information" refers to data used to identify a caller, such as a phone number or other unique information.

[0317] A "database" is a collection of data that stores past communication history and evaluations, and is used as reference information for calculating reliability.

[0318] "Trustworthiness" is an indicator used to evaluate the sender's communication behavior and demonstrate the security and legitimacy of the communication.

[0319] "Real-time" is a term that indicates that information acquisition and processing occur virtually instantaneously.

[0320] "Anomaly" refers to behavior that deviates from normal communication patterns or suspicious points that suggest a risk of fraud.

[0321] This invention relates to a system for enabling safe and reliable communication for users, including the elderly and those with dementia. A server functions as the central component of this system and is implemented as follows:

[0322] When the server receives an incoming call, it automatically provides a message to the caller using a voice response system and collects the necessary audio data. This audio data is converted to text data in real time on the terminal. Software such as TensorFlow or natural language processing libraries (e.g., spaCy) are used for the audio-to-text conversion.

[0323] Furthermore, the server matches the caller's identification information against a database and refers to past communication history in the database to calculate the reliability score. This reliability score is updated in real time, enhancing the security of calls.

[0324] The user's device has a feature that only notifies the user of incoming calls if the call's trustworthiness exceeds a certain threshold. This ensures that the user only receives calls from trusted sources. If a suspicious pattern is detected, the device will alert the user. This strengthens protection, especially against potentially fraudulent calls.

[0325] For example, if a user receives a call claiming to be from the city hall with an "important announcement," the server will automatically block the call if the caller's trustworthiness is low. This also helps the user prevent becoming a victim of fraud. To standardize this response process, the following is used as an example of a prompt:

[0326] "Input: Convert the recorded audio of a phone call from the city hall into text and analyze it for any suspicious patterns."

[0327] "Output: Confidence score and warning message for suspicious patterns"

[0328] Thus, the system according to the present invention aims to provide a safe and smooth communication experience for the elderly and dementia patients.

[0329] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0330] Step 1:

[0331] When the server receives an incoming call, it activates the voice response system and automatically sends a message to the caller. The input is an incoming call, and the output is a voice message to the caller. This process captures the caller's voice data.

[0332] Step 2:

[0333] The device converts acquired audio data into text data in real time. The input is the acquired audio data, and the output is the corresponding text data. Speech recognition technology is used here; for example, TensorFlow is used to convert audio to text.

[0334] Step 3:

[0335] The server applies natural language processing to text data to create a summary. The input is text data, and the output is summarized text. Here, a generative AI model is used to summarize the data, structuring the information.

[0336] Step 4:

[0337] The server compares the caller's identification information with a database and calculates the confidence score. The input is the caller's identification information and the database used for comparison, and the output is the confidence score. This confidence score is used to evaluate the caller's safety.

[0338] Step 5:

[0339] The server decides whether to allow or deny a call based on a confidence score. The input is the confidence score, and the output is whether the call is allowed or denied. Based on the set threshold, the call is only allowed if the caller is deemed trustworthy.

[0340] Step 6:

[0341] If a call is permitted, the user's device analyzes the call content in real time and monitors for any abnormalities. The input is the call content, and the output is whether or not there are any abnormalities. This monitoring allows the user to confirm that they are safe even while on a call.

[0342] Step 7:

[0343] If an anomaly is detected, the device will immediately issue a warning to the user. The input is the analyzed call content, and the output is a warning notification. Specifically, if a suspicious request is detected, the device will alert the user by displaying a warning sound or notification.

[0344] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0345] This invention is a system that combines an emotion engine for recognizing user emotions with an incoming call management system in a communication device. Specific embodiments of this system are described below.

[0346] When the server detects an incoming call, it activates the voice response system and requests a voice response from the caller. The caller's response is received as voice data by the terminal and converted to text. This text data is summarized using natural language processing techniques, and the caller's name and request are analyzed.

[0347] Simultaneously, the server compares the caller's identification information with a database and calculates a confidence score based on past communication history and evaluations. Based on this confidence score, the server decides whether to allow the user to make a call.

[0348] Furthermore, during a call, the device's built-in emotion engine analyzes the user's voice tone, speaking speed, and word choice to recognize emotions in real time. The emotion engine detects basic emotions such as joy, anger, surprise, sadness, and anxiety, and supports decision-making based on the communication situation. Based on this information, the server can decide whether to continue the call, interrupt it, or issue additional warnings.

[0349] As a concrete example, consider a scenario where a user answers a call and the caller begins speaking in an aggressive tone. The emotion engine recognizes this as "anger," and if the user starts to feel stressed, the device can emit a warning sound or display a message on the screen asking, "Do you want to continue the call?" There is also an option to automatically terminate the call if the emotional escalation occurs.

[0350] Thus, by combining emotion analysis using an emotion engine and realizing even more advanced call management, the present invention can provide users with a safer and more comfortable calling environment.

[0351] The following describes the processing flow.

[0352] Step 1:

[0353] When the server detects an incoming call, it activates the voice response system. The server automatically sends a voice message to the caller saying, "Please tell me your name and how can I help you?"

[0354] Step 2:

[0355] When the caller answers, the terminal acquires the audio data and converts it into text data in real time. Speech recognition technology is used in this process.

[0356] Step 3:

[0357] The terminal feeds the text data into a natural language processing engine to generate a summary. This summarizes the caller's name and the main points of the request.

[0358] Step 4:

[0359] The server checks the caller's phone number against a database. The database contains records of past communication history and ratings, and the server calculates a confidence score based on this information.

[0360] Step 5:

[0361] The server evaluates the confidence score and decides whether to ring the user's device. If the score exceeds a certain threshold, the device will ring to inform the user of the purpose of the call. Otherwise, the call will be automatically rejected or switched to voicemail mode.

[0362] Step 6:

[0363] When a user initiates a call, the device activates its emotion engine. The emotion engine analyzes the user's voice tone and speaking speed in real time and evaluates their emotional state.

[0364] Step 7:

[0365] When the emotion engine detects stressful states such as "anger" or "anxiety," the device issues a warning to the user. Specifically, it sounds a warning and displays a warning message on the screen.

[0366] Step 8:

[0367] In response to the device's warning, the user can choose to continue or end the call. The system can also automatically terminate the call if there are signs of emotional escalation.

[0368] Through this series of processing steps, the system provides users with a safe and comfortable calling environment.

[0369] (Example 2)

[0370] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0371] Modern communication devices are required to appropriately assess caller information and emotional state during calls to provide a safe and comfortable calling experience. However, conventional systems have not adequately verified caller information or analyzed emotions during calls, potentially leading to unpleasant experiences for users. Therefore, the challenge is to provide technology that enables appropriate call management based on caller reliability and emotional changes during calls.

[0372] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0373] In this invention, the server includes means for activating a voice response function when a communication device detects an incoming call and obtaining voice information from the caller; means for converting the voice information into text information and summarizing it using information processing technology; means for comparing the caller's identification information with a recording device and calculating the reliability level; means for analyzing emotions in real time during a call using an information analysis engine and providing warnings according to those emotions; and means for automatically terminating a call when emotions exceed a certain threshold. This enables users to have a safe and comfortable calling environment.

[0374] "Communication equipment" refers to devices used to send and receive voice and data, and includes telephones, smartphones, and network devices.

[0375] "Voice response function" refers to a system in which a communication device automatically receives voice messages and generates the necessary response.

[0376] "Audio information" refers to data transmitted through human voices, and includes phone calls and recorded audio.

[0377] "Text information" refers to information expressed as a string of characters, and includes data obtained by converting audio information into text.

[0378] "Information processing technology" refers to computer technologies used to analyze, transform, and utilize data, and includes natural language processing, among others.

[0379] "Identification information" refers to data used to identify a specific individual or sender, and includes names and ID numbers.

[0380] A "recording device" refers to a device for storing information, and this includes databases and storage systems.

[0381] "Trustworthiness" is a value that indicates how trustworthy a particular piece of information or its source is, and is calculated based on past history and evaluations.

[0382] An "information analysis engine" refers to software or algorithms used to analyze data and extract meaning and patterns.

[0383] "Emotions" refer to the psychological state exhibited by the caller and receiver during a phone call, and include feelings such as joy, anger, surprise, sadness, and anxiety.

[0384] A "standard" refers to a benchmark or reference point used when making a particular action or judgment.

[0385] This invention combines an emotion recognition engine with an incoming call management system for communication devices, providing a system that enables users to have safer and more comfortable phone calls. This system mainly consists of a server and terminals, each performing its own specific role.

[0386] The server's processing begins when the communication device detects an incoming call; it activates the voice response function to obtain voice information from the caller. The voice response system uses any speech synthesis technology to request the caller to provide their name and purpose of the call. The acquired voice information is then converted into text on the server. For speech recognition, commonly used technologies such as the Google Speech-to-Text API or open-source speech recognition libraries can be utilized.

[0387] The server analyzes the converted text information using information processing technology and calculates a confidence score by comparing the caller's identification information with the recording device. This can be done by applying natural language processing technology and incorporating libraries such as Apache OpenNLP and Stanford NLP. The confidence score thus calculated is used as a criterion for deciding whether to allow or deny the call.

[0388] During a call, the device uses an information analysis engine to analyze the user's emotions in real time. This emotion engine employs technologies such as Affectiva and Emotion AI to identify emotions from changes in the user's voice tone and speaking style. This emotion data is sent to a server and contributes to issuing warnings based on the communication status and determining when to end the call.

[0389] As a concrete example, consider a scenario where the caller's voice becomes aggressive during a call. When the emotion engine recognizes "anger," the device emits a warning sound and displays the option "Do you wish to continue the call?" on the screen. Furthermore, a feature is provided that allows the user to automatically terminate the call if such emotions escalate.

[0390] A concrete example of a prompt for a generative AI model could be: "What is the most effective way to warn a user when they are receiving a call from someone speaking in an aggressive tone?"

[0391] This system allows users to enjoy a more comfortable and secure calling environment based on the caller's trustworthiness and emotional state during the call.

[0392] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0393] Step 1:

[0394] The server detects an incoming call from the communication device and activates the voice response function. The server receives the incoming signal as input and issues a command to activate the voice response system. This process prepares the server to begin providing a voice message to the caller.

[0395] Step 2:

[0396] The server obtains voice information from the caller. The voice response system receives the caller's response in voice format and sends it to the server. The input is the caller's raw voice data, which is stored on the server for later processing.

[0397] Step 3:

[0398] The server converts the audio information into text information. The server uses speech recognition software to process the acquired audio data and generate text data. This conversion makes the audio content available as text information.

[0399] Step 4:

[0400] The server analyzes and summarizes text information. Utilizing natural language processing technology, it processes the input text information to extract important elements such as the caller's name and purpose. The output is summarized information, which can be used for call management.

[0401] Step 5:

[0402] The server compares the caller's identification information with the recording device and calculates the confidence level. The server takes caller identification data as input and evaluates the confidence level by comparing it with a database. The confidence level calculated based on past communication history is stored as output within the server.

[0403] Step 6:

[0404] The server decides whether to allow or deny the call based on its trustworthiness. Based on the acquired trustworthiness, the server makes a decision: if it allows the call, it continues the call; if it denies it, it terminates the call. The output is determined as either the allowed or denied status of the call.

[0405] Step 7:

[0406] The device analyzes the user's emotions in real time during a call. The device uses a data analysis engine to identify emotions based on the user's recorded voice. The recognized emotion information is sent to a server and used to inform communication decisions.

[0407] Step 8:

[0408] The server provides users with emotionally-sensitive warnings. Based on emotional information, the server displays warning sounds or messages to the user via the terminal if necessary, ensuring the safety of the call. It receives emotional information as input and utilizes warning methods as output.

[0409] Step 9:

[0410] The server automatically terminates the call when emotions exceed a certain threshold. The server compares emotional information with the set threshold and, if the threshold is exceeded, automatically outputs a command to end the call. This action helps users avoid unnecessary stress and unpleasant situations.

[0411] (Application Example 2)

[0412] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0413] There is a need to improve the safety and comfort of calls using communication devices. In particular, there is a lack of means to recognize the caller's emotions and stress levels in real time, which presents a challenge in responding appropriately to situations where users feel anxious or stressed. In this situation, an effective system is needed to mitigate the risk of users encountering unforeseen circumstances.

[0414] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0415] In this invention, the server includes means for activating a voice response system when a communication device receives an incoming call and acquiring voice data from the caller, means for analyzing voice tone and speaking speed to recognize emotions in real time, and means for issuing an alert when the user feels stressed. This makes it possible to analyze the caller's emotional state and provide appropriate feedback to the user.

[0416] A "communication device" is a device that transmits and receives voice data, enabling communication between users.

[0417] A "voice response system" is a system that automatically activates when a call is received and receives voice data from the caller.

[0418] "Audio data" refers to data that records the speaker's utterance as a digital signal.

[0419] "Natural language processing" is an information processing technology that analyzes text data and derives its meaning and summary.

[0420] "Caller identification information" refers to information used to identify a caller, and includes information such as a phone number and name.

[0421] "Trustworthiness" is an indicator of whether a sender is trustworthy, and it is a score based on past communication history and evaluations.

[0422] "Voice tone" refers to the characteristics of pitch and intensity of a voice that reflect the speaker's psychological state and emotions.

[0423] "Speech rate" is an indicator that shows the number of words a speaker utters within a certain period of time, and is a standard for measuring the pace of communication.

[0424] "Means of recognizing emotions in real time" refers to technologies and methods for instantly determining the emotional state of a speaker while they are speaking.

[0425] "Means of issuing alerts" refers to notification functions used to warn or alert users.

[0426] To realize this invention, it is necessary to install a dedicated application on communication devices such as smartphones and servers. The main functions of the system are voice data processing and emotion recognition, and it operates as follows.

[0427] First, when a communication device receives an incoming call, the server activates the voice response system and retrieves the caller's voice data. This data is converted into text data using a speech recognition API (e.g., Google Cloud Speech-to-Text). Next, the text is summarized using a natural language processing library (e.g., spaCy), and the caller's identification information and confidence level are calculated through database matching.

[0428] The server further analyzes the received voice tone and speaking speed in real time using an emotion recognition engine built into the device. This analysis detects the caller's emotions, such as joy or anger. If the user feels stressed, the device issues an alert and presents the user with options such as "Do you want to continue the call?". It can also automatically terminate the call if the emotions escalate.

[0429] For example, if the system detects unsettling emotions during a call, it will automatically trigger a prompt in real time to "monitor the caller's emotions and notify the user if their tone is unsettling." This mechanism protects users from unexpected and unpleasant interactions.

[0430] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0431] Step 1:

[0432] The server activates the voice response system when the communication device receives an incoming call and acquires voice data from the caller. The input at this time is a real-time voice signal, and the output is the acquired voice data. The voice response system uses a microphone to capture the voice signal.

[0433] Step 2:

[0434] The server converts the acquired audio data into text data using a speech recognition API. The input is audio data, and the output is the corresponding text data. A speech recognition service such as Google Cloud Speech-to-Text is used for this conversion.

[0435] Step 3:

[0436] The server analyzes text data using a natural language processing library and summarizes the sender's requirements. The input is text data, and the output is the summarized content. A natural language processing tool like spaCy extracts keywords and context through the analysis.

[0437] Step 4:

[0438] The server compares the caller's identification information with a database and calculates the confidence score. The input is the caller's identification information, and the output is the caller's confidence score. The calculation is based on past communication history and evaluations.

[0439] Step 5:

[0440] The server decides whether to allow or deny a call based on a confidence score. The input is the confidence score, and the output is whether the call is allowed or denied. If the confidence score falls below a certain threshold, the call may be denied.

[0441] Step 6:

[0442] The server uses the emotion recognition engine built into the terminal to analyze the voice tone and speech rate during communication in real time. The input is text data and voice characteristics, and the output is the emotion analysis result. A generative AI model detects emotions and determines specific emotional states such as "joy" or "anger."

[0443] Step 7:

[0444] The device issues an alert when the user experiences stress. The input is the result of sentiment analysis, and the output is an alert to the user. The alert is delivered to the user as a visual or audio notification.

[0445] Step 8:

[0446] An interface is provided for the user to choose a course of action. The input is the user's selection, and the output is an action such as continuing or ending the call. A prompt such as "Do you want to continue the call?" is displayed to the user.

[0447] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0448] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0449] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0450] [Third Embodiment]

[0451] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0452] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0453] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0454] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0455] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0456] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0457] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0458] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0459] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0460] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0461] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0462] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0463] This invention provides a system that enables users, including the elderly and those with dementia, to engage in safe and reliable communication. Specific embodiments of the system are described below.

[0464] In this invention, the server acts as a communication device, receiving all incoming calls first. Upon receiving an incoming call, the server activates an AI-powered voice response system and automatically sends a message to the caller saying, "Please tell us your name and purpose of your call." The caller's voice response is converted into text data in real time by the terminal.

[0465] The terminal summarizes this text data using natural language processing technology and stores it in a structured format. Next, the server matches the caller's identification information (e.g., phone number) against a dedicated database. This database stores information on past communication history and the trustworthiness of each caller, and a trust score for the caller is calculated based on this information.

[0466] The confidence score serves as a criterion for the server to decide whether to allow a call. If the confidence score exceeds a certain level, the server will make the user's device ring to inform the user of the intention to call. This allows the user to receive only calls from trusted callers. On the other hand, if the confidence score is low, the server will automatically terminate the call or switch to voicemail mode.

[0467] Even after the user answers a call, the device continues to monitor the call content in real time. If any unusual patterns or content that does not match the initial summary is detected during the call, the device can alert the user. For example, if an inappropriate request suddenly appears, the device will automatically sound an alert. This feature allows users to make calls with greater peace of mind.

[0468] A concrete example is a case where a server receives an incoming call, and the caller, claiming to be a "financial institution representative," attempts to verify account information. The server checks the caller's number in its database and calculates its reliability by referring to past records. If the number matches one suspected of being a fraudulent call, the server rejects the call and does not notify the user. In this way, the present invention prevents fraudulent communications and protects users securely.

[0469] The following describes the processing flow.

[0470] Step 1:

[0471] When the server detects an incoming call, it activates the voice response system. It automatically sends a message to the caller saying, "Please tell me your name and how can I help you?"

[0472] Step 2:

[0473] When the caller responds to a message, the device converts the audio data into text data in real time. Speech recognition technology is used to convert the caller's speech into digital text information.

[0474] Step 3:

[0475] The text data acquired by the terminal is fed into a natural language processing engine for summarization. This generates a concise summary of the caller's name and the purpose of the message.

[0476] Step 4:

[0477] The server checks the caller's phone number against a database and calculates a confidence score based on past communication history and ratings. It utilizes information from reliable data sources to quantify the caller's trustworthiness.

[0478] Step 5:

[0479] The server evaluates the trust score and decides whether to ring the user's device. If the trust score exceeds the threshold, the device rings to allow the user to make a call. If the trust score is low, the call is automatically rejected or switched to voicemail mode.

[0480] Step 6:

[0481] When a user starts a call, the device monitors the call content in real time. It transcribes the audio data during the call and verifies whether it matches the initial summary.

[0482] Step 7:

[0483] If unusual content or inconsistencies with the initial summary are detected during a call, the device will alert the user. If necessary, it will automatically terminate the call to protect the user.

[0484] By following the steps outlined above, this system can provide users with a safe and reliable calling environment.

[0485] (Example 1)

[0486] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0487] The goal is to provide a system that allows users, including the elderly and those with dementia, to communicate safely and securely when they receive fraudulent calls or unauthorized communications. Conventional communication methods make it difficult to accurately determine the caller's intentions, necessitating appropriate security measures. This system aims to prevent confusion and problems caused by incoming calls from unreliable callers.

[0488] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0489] In this invention, the server includes means for activating a voice response function using a generative model when a communication device receives an incoming call, presenting a prompt message to the caller to obtain voice information, converting the voice information into text and summarizing it using a natural language processing function, and comparing the caller's identification number with an information base to calculate a reliability rating. As a result, users can receive only communications from trusted callers, eliminate abnormal communications in advance, and maintain a safe and secure communication environment.

[0490] A "communication device" is a device that can send and receive voice and data, and has the function of starting the next processing step upon receiving an incoming call.

[0491] A "generative model" refers to an algorithm that operates using artificial intelligence, understands human language, and generates responses, and is used to drive voice response functions.

[0492] A "prompt message" is a message designed to prompt the caller for information or a response, and refers to questions or instructions automatically generated by voice response functions.

[0493] "Audio information" refers to audio data emitted by a caller, which is converted into text through speech recognition processing.

[0494] "Natural language processing" refers to the technology that allows computers to understand, interpret, and generate human language, and is used to summarize text information.

[0495] An "identification value" is numerical data used to identify the sender and verify the origin of the message.

[0496] An "information base" refers to a data storage system that accumulates past communication history and sender reliability information, and is used as a criterion for reliability evaluation.

[0497] "Trust rating" is a numerical representation of the trustworthiness of a sender, calculated based on their past history and evaluations.

[0498] "Call information" refers to the audio and text data generated during a call, and is the content of the communication that is subject to analysis.

[0499] This invention provides a system that enables users, including the elderly and those with dementia, to communicate safely and reliably. The system is comprised of a combination of voice response functionality, speech recognition technology, natural language processing technology, identification information matching, reliability evaluation, and call monitoring.

[0500] The server operates as a communication device and receives incoming calls from callers. Upon receiving a call, the server uses a generative AI model to activate its voice response function. This function generates a prompt message and presents it to the caller. An example of a prompt message used is, "Please tell us your name and purpose of your call." This prompt message serves to encourage the caller to provide a clear response.

[0501] The device utilizes speech recognition technology to convert the caller's voice information into text in real time. Specifically, general-purpose speech recognition software is used. The converted text data is summarized using natural language processing techniques. This allows the caller's requirements and intentions to be clarified in a short amount of time.

[0502] The server compares the caller's identification number with an information base and calculates a trust rating. This information base stores past communication history and ratings of the caller's trustworthiness. Based on this trust rating, the server decides whether to allow or deny the call.

[0503] Even if the user answers, the terminal continues to analyze the call information, and if it detects suspicious patterns or content that does not match the initial summary, it will warn the user. In this way, users can communicate securely and reliably.

[0504] A concrete example is a case where the caller impersonates a financial institution employee and attempts to verify account information. In this case, the server checks the caller's identification number and rejects the call if it is suspected to be fraudulent. At the same time, the user receives a warning about the suspicious content. This system prevents fraudulent communication and ensures the safety of users.

[0505] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0506] Step 1:

[0507] The server receives the incoming call. The input is an audio signal along with the caller's identifier. This incoming signal invokes the voice response function. The output is a preparation to issue a prompt to the caller using a generative AI model. This prompt is "Please tell us your name and purpose of call."

[0508] Step 2:

[0509] The device converts the caller's voice response into text data in real time. It receives the voice signal as input and digitizes it as text data using speech recognition technology. The output is text data in string format. Specifically, the content spoken by the caller is accurately transcribed into text.

[0510] Step 3:

[0511] The terminal summarizes this text data using natural language processing techniques. The input is the text data obtained in step 2. A natural language processing algorithm is applied to this data to extract important information and generate a simplified summary. The output is a summarized text, making the sender's requirements easier to understand.

[0512] Step 4:

[0513] The server compares the caller's identification number against a dedicated information base. The input is the caller identification data received in step 1. By comparing it with past communication history stored in the information base, the server calculates a trust rating for the caller. The output is a trust score. Based on this score, the server is ready to decide whether to allow or deny the call.

[0514] Step 5:

[0515] The server decides how to handle the call based on the trust score. The input is the trust score calculated in step 4. If the trust score is above a certain level, the server rings the user's device and allows the call. Conversely, if the trust score is low, the server rejects the call or switches to voicemail mode. The output is whether the call was allowed or terminated.

[0516] Step 6:

[0517] When a user receives a call, the device monitors the call content. It converts the audio data into text in real time and verifies that it matches the initial summary. The input is the audio data generated during the call. As output, it warns the user if any unusual patterns are detected. Specifically, an alert will sound if a sudden, inappropriate request occurs.

[0518] (Application Example 1)

[0519] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0520] The communication risks that users, including the elderly and those with dementia, face on a daily basis can cause serious problems, particularly through fraudulent attempts such as scams. This invention aims to prevent such fraudulent communications and provide a safe and reliable calling experience. In particular, it is necessary to provide a method for detecting fraudulent calls in real time and preventing harm to users.

[0521] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0522] In this invention, the server includes means for activating a voice response system when a communication device receives an incoming call and acquiring voice data from the caller; means for converting the voice data into text data and summarizing it using natural language processing; means for comparing the caller's identification information with a database and calculating the reliability level; and means for re-evaluating the reliability level in real time and allowing only secure calls. This makes it possible to detect fraudulent communications as suspicious patterns and issue warnings to users.

[0523] "Communication equipment" refers to devices and means for receiving incoming calls, and plays a central role in acquiring and processing voice data.

[0524] A "voice response system" is a program or function that automatically provides a voice message to the caller upon receiving a call and collects necessary information.

[0525] "Audio data" refers to audio information obtained from the caller, and is subject to analysis and text conversion.

[0526] "Text data" refers to the written information that remains after audio data has been converted, and is a data format used for natural language processing and summarization.

[0527] "Natural language processing" refers to the technology of analyzing and processing human language using computers, and is a technical means of summarizing text data.

[0528] "Caller identification information" refers to data used to identify a caller, such as a phone number or other unique information.

[0529] A "database" is a collection of data that stores past communication history and evaluations, and is used as reference information for calculating reliability.

[0530] "Trustworthiness" is an indicator used to evaluate the sender's communication behavior and demonstrate the security and legitimacy of the communication.

[0531] "Real-time" is a term that indicates that information acquisition and processing occur virtually instantaneously.

[0532] "Anomaly" refers to behavior that deviates from normal communication patterns or suspicious points that suggest a risk of fraud.

[0533] This invention relates to a system for enabling safe and reliable communication for users, including the elderly and those with dementia. A server functions as the central component of this system and is implemented as follows:

[0534] When the server receives an incoming call, it automatically provides a message to the caller using a voice response system and collects the necessary audio data. This audio data is converted to text data in real time on the terminal. Software such as TensorFlow or natural language processing libraries (e.g., spaCy) are used for the audio-to-text conversion.

[0535] Furthermore, the server matches the caller's identification information against a database and refers to past communication history in the database to calculate the reliability score. This reliability score is updated in real time, enhancing the security of calls.

[0536] The user's device has a feature that only notifies the user of incoming calls if the call's trustworthiness exceeds a certain threshold. This ensures that the user only receives calls from trusted sources. If a suspicious pattern is detected, the device will alert the user. This strengthens protection, especially against potentially fraudulent calls.

[0537] For example, if a user receives a call claiming to be from the city hall with an "important announcement," the server will automatically block the call if the caller's trustworthiness is low. This also helps the user prevent becoming a victim of fraud. To standardize this response process, the following is used as an example of a prompt:

[0538] "Input: Convert the recorded audio of a phone call from the city hall into text and analyze it for any suspicious patterns."

[0539] "Output: Confidence score and warning message for suspicious patterns"

[0540] Thus, the system according to the present invention aims to provide a safe and smooth communication experience for the elderly and dementia patients.

[0541] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0542] Step 1:

[0543] When the server receives an incoming call, it activates the voice response system and automatically sends a message to the caller. The input is an incoming call, and the output is a voice message to the caller. This process captures the caller's voice data.

[0544] Step 2:

[0545] The device converts acquired audio data into text data in real time. The input is the acquired audio data, and the output is the corresponding text data. Speech recognition technology is used here; for example, TensorFlow is used to convert audio to text.

[0546] Step 3:

[0547] The server applies natural language processing to text data to create a summary. The input is text data, and the output is summarized text. Here, a generative AI model is used to summarize the data, structuring the information.

[0548] Step 4:

[0549] The server compares the caller's identification information with a database and calculates the confidence score. The input is the caller's identification information and the database used for comparison, and the output is the confidence score. This confidence score is used to evaluate the caller's safety.

[0550] Step 5:

[0551] The server decides whether to allow or deny a call based on a confidence score. The input is the confidence score, and the output is whether the call is allowed or denied. Based on the set threshold, the call is only allowed if the caller is deemed trustworthy.

[0552] Step 6:

[0553] If a call is permitted, the user's device analyzes the call content in real time and monitors for any abnormalities. The input is the call content, and the output is whether or not there are any abnormalities. This monitoring allows the user to confirm that they are safe even while on a call.

[0554] Step 7:

[0555] If an anomaly is detected, the device will immediately issue a warning to the user. The input is the analyzed call content, and the output is a warning notification. Specifically, if a suspicious request is detected, the device will alert the user by displaying a warning sound or notification.

[0556] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0557] This invention is a system that combines an emotion engine for recognizing user emotions with an incoming call management system in a communication device. Specific embodiments of this system are described below.

[0558] When the server detects an incoming call, it activates the voice response system and requests a voice response from the caller. The caller's response is received as voice data by the terminal and converted to text. This text data is summarized using natural language processing techniques, and the caller's name and request are analyzed.

[0559] Simultaneously, the server compares the caller's identification information with a database and calculates a confidence score based on past communication history and evaluations. Based on this confidence score, the server decides whether to allow the user to make a call.

[0560] Furthermore, during a call, the device's built-in emotion engine analyzes the user's voice tone, speaking speed, and word choice to recognize emotions in real time. The emotion engine detects basic emotions such as joy, anger, surprise, sadness, and anxiety, and supports decision-making based on the communication situation. Based on this information, the server can decide whether to continue the call, interrupt it, or issue additional warnings.

[0561] As a concrete example, consider a scenario where a user answers a call and the caller begins speaking in an aggressive tone. The emotion engine recognizes this as "anger," and if the user starts to feel stressed, the device can emit a warning sound or display a message on the screen asking, "Do you want to continue the call?" There is also an option to automatically terminate the call if the emotional escalation occurs.

[0562] Thus, by combining emotion analysis using an emotion engine and realizing even more advanced call management, the present invention can provide users with a safer and more comfortable calling environment.

[0563] The following describes the processing flow.

[0564] Step 1:

[0565] When the server detects an incoming call, it activates the voice response system. The server automatically sends a voice message to the caller saying, "Please tell me your name and how can I help you?"

[0566] Step 2:

[0567] When the caller answers, the terminal acquires the audio data and converts it into text data in real time. Speech recognition technology is used in this process.

[0568] Step 3:

[0569] The terminal feeds the text data into a natural language processing engine to generate a summary. This summarizes the caller's name and the main points of the request.

[0570] Step 4:

[0571] The server checks the caller's phone number against a database. The database contains records of past communication history and ratings, and the server calculates a confidence score based on this information.

[0572] Step 5:

[0573] The server evaluates the confidence score and decides whether to ring the user's device. If the score exceeds a certain threshold, the device will ring to inform the user of the purpose of the call. Otherwise, the call will be automatically rejected or switched to voicemail mode.

[0574] Step 6:

[0575] When a user initiates a call, the device activates its emotion engine. The emotion engine analyzes the user's voice tone and speaking speed in real time and evaluates their emotional state.

[0576] Step 7:

[0577] When the emotion engine detects stressful states such as "anger" or "anxiety," the device issues a warning to the user. Specifically, it sounds a warning and displays a warning message on the screen.

[0578] Step 8:

[0579] In response to the device's warning, the user can choose to continue or end the call. The system can also automatically terminate the call if there are signs of emotional escalation.

[0580] Through this series of processing steps, the system provides users with a safe and comfortable calling environment.

[0581] (Example 2)

[0582] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0583] Modern communication devices are required to appropriately assess caller information and emotional state during calls to provide a safe and comfortable calling experience. However, conventional systems have not adequately verified caller information or analyzed emotions during calls, potentially leading to unpleasant experiences for users. Therefore, the challenge is to provide technology that enables appropriate call management based on caller reliability and emotional changes during calls.

[0584] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0585] In this invention, the server includes means for activating a voice response function when a communication device detects an incoming call and obtaining voice information from the caller; means for converting the voice information into text information and summarizing it using information processing technology; means for comparing the caller's identification information with a recording device and calculating the reliability level; means for analyzing emotions in real time during a call using an information analysis engine and providing warnings according to those emotions; and means for automatically terminating a call when emotions exceed a certain threshold. This enables users to have a safe and comfortable calling environment.

[0586] "Communication equipment" refers to devices used to send and receive voice and data, and includes telephones, smartphones, and network devices.

[0587] "Voice response function" refers to a system in which a communication device automatically receives voice messages and generates the necessary response.

[0588] "Audio information" refers to data transmitted through human voices, and includes phone calls and recorded audio.

[0589] "Text information" refers to information expressed as a string of characters, and includes data obtained by converting audio information into text.

[0590] "Information processing technology" refers to computer technologies used to analyze, transform, and utilize data, and includes natural language processing, among others.

[0591] "Identification information" refers to data used to identify a specific individual or sender, and includes names and ID numbers.

[0592] A "recording device" refers to a device for storing information, and this includes databases and storage systems.

[0593] "Trustworthiness" is a value that indicates how trustworthy a particular piece of information or its source is, and is calculated based on past history and evaluations.

[0594] An "information analysis engine" refers to software or algorithms used to analyze data and extract meaning and patterns.

[0595] "Emotions" refer to the psychological state exhibited by the caller and receiver during a phone call, and include feelings such as joy, anger, surprise, sadness, and anxiety.

[0596] A "standard" refers to a benchmark or reference point used when making a particular action or judgment.

[0597] This invention combines an emotion recognition engine with an incoming call management system for communication devices, providing a system that enables users to have safer and more comfortable phone calls. This system mainly consists of a server and terminals, each performing its own specific role.

[0598] The server's processing begins when the communication device detects an incoming call; it activates the voice response function to obtain voice information from the caller. The voice response system uses any speech synthesis technology to request the caller to provide their name and purpose of the call. The acquired voice information is then converted into text on the server. For speech recognition, commonly used technologies such as the Google Speech-to-Text API or open-source speech recognition libraries can be utilized.

[0599] The server analyzes the converted text information using information processing technology and calculates a confidence score by comparing the caller's identification information with the recording device. This can be done by applying natural language processing technology and incorporating libraries such as Apache OpenNLP and Stanford NLP. The confidence score thus calculated is used as a criterion for deciding whether to allow or deny the call.

[0600] During a call, the device uses an information analysis engine to analyze the user's emotions in real time. This emotion engine employs technologies such as Affectiva and Emotion AI to identify emotions from changes in the user's voice tone and speaking style. This emotion data is sent to a server and contributes to issuing warnings based on the communication status and determining when to end the call.

[0601] As a concrete example, consider a scenario where the caller's voice becomes aggressive during a call. When the emotion engine recognizes "anger," the device emits a warning sound and displays the option "Do you wish to continue the call?" on the screen. Furthermore, a feature is provided that allows the user to automatically terminate the call if such emotions escalate.

[0602] A concrete example of a prompt for a generative AI model could be: "What is the most effective way to warn a user when they are receiving a call from someone speaking in an aggressive tone?"

[0603] This system allows users to enjoy a more comfortable and secure calling environment based on the caller's trustworthiness and emotional state during the call.

[0604] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0605] Step 1:

[0606] The server detects an incoming call from the communication device and activates the voice response function. The server receives the incoming signal as input and issues a command to activate the voice response system. This process prepares the server to begin providing a voice message to the caller.

[0607] Step 2:

[0608] The server obtains voice information from the caller. The voice response system receives the caller's response in voice format and sends it to the server. The input is the caller's raw voice data, which is stored on the server for later processing.

[0609] Step 3:

[0610] The server converts the audio information into text information. The server uses speech recognition software to process the acquired audio data and generate text data. This conversion makes the audio content available as text information.

[0611] Step 4:

[0612] The server analyzes and summarizes text information. Utilizing natural language processing technology, it processes the input text information to extract important elements such as the caller's name and purpose. The output is summarized information, which can be used for call management.

[0613] Step 5:

[0614] The server compares the caller's identification information with the recording device and calculates the confidence level. The server takes caller identification data as input and evaluates the confidence level by comparing it with a database. The confidence level calculated based on past communication history is stored as output within the server.

[0615] Step 6:

[0616] The server decides whether to allow or deny the call based on its trustworthiness. Based on the acquired trustworthiness, the server makes a decision: if it allows the call, it continues the call; if it denies it, it terminates the call. The output is determined as either the allowed or denied status of the call.

[0617] Step 7:

[0618] The device analyzes the user's emotions in real time during a call. The device uses a data analysis engine to identify emotions based on the user's recorded voice. The recognized emotion information is sent to a server and used to inform communication decisions.

[0619] Step 8:

[0620] The server provides users with emotionally-sensitive warnings. Based on emotional information, the server displays warning sounds or messages to the user via the terminal if necessary, ensuring the safety of the call. It receives emotional information as input and utilizes warning methods as output.

[0621] Step 9:

[0622] The server automatically terminates the call when emotions exceed a certain threshold. The server compares emotional information with the set threshold and, if the threshold is exceeded, automatically outputs a command to end the call. This action helps users avoid unnecessary stress and unpleasant situations.

[0623] (Application Example 2)

[0624] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0625] There is a need to improve the safety and comfort of calls using communication devices. In particular, there is a lack of means to recognize the caller's emotions and stress levels in real time, which presents a challenge in responding appropriately to situations where users feel anxious or stressed. In this situation, an effective system is needed to mitigate the risk of users encountering unforeseen circumstances.

[0626] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0627] In this invention, the server includes means for activating a voice response system when a communication device receives an incoming call and acquiring voice data from the caller, means for analyzing voice tone and speaking speed to recognize emotions in real time, and means for issuing an alert when the user feels stressed. This makes it possible to analyze the caller's emotional state and provide appropriate feedback to the user.

[0628] A "communication device" is a device that transmits and receives voice data, enabling communication between users.

[0629] A "voice response system" is a system that automatically activates when a call is received and receives voice data from the caller.

[0630] "Audio data" refers to data that records the speaker's utterance as a digital signal.

[0631] "Natural language processing" is an information processing technology that analyzes text data and derives its meaning and summary.

[0632] "Caller identification information" refers to information used to identify a caller, and includes information such as a phone number and name.

[0633] "Trustworthiness" is an indicator of whether a sender is trustworthy, and it is a score based on past communication history and evaluations.

[0634] "Voice tone" refers to the characteristics of pitch and intensity of a voice that reflect the speaker's psychological state and emotions.

[0635] "Speech rate" is an indicator that shows the number of words a speaker utters within a certain period of time, and is a standard for measuring the pace of communication.

[0636] "Means of recognizing emotions in real time" refers to technologies and methods for instantly determining the emotional state of a speaker while they are speaking.

[0637] "Means of issuing alerts" refers to notification functions used to warn or alert users.

[0638] To realize this invention, it is necessary to install a dedicated application on communication devices such as smartphones and servers. The main functions of the system are voice data processing and emotion recognition, and it operates as follows.

[0639] First, when a communication device receives an incoming call, the server activates the voice response system and retrieves the caller's voice data. This data is converted into text data using a speech recognition API (e.g., Google Cloud Speech-to-Text). Next, the text is summarized using a natural language processing library (e.g., spaCy), and the caller's identification information and confidence level are calculated through database matching.

[0640] The server further analyzes the received voice tone and speaking speed in real time using an emotion recognition engine built into the device. This analysis detects the caller's emotions, such as joy or anger. If the user feels stressed, the device issues an alert and presents the user with options such as "Do you want to continue the call?". It can also automatically terminate the call if the emotions escalate.

[0641] For example, if the system detects unsettling emotions during a call, it will automatically trigger a prompt in real time to "monitor the caller's emotions and notify the user if their tone is unsettling." This mechanism protects users from unexpected and unpleasant interactions.

[0642] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0643] Step 1:

[0644] The server activates the voice response system when the communication device receives an incoming call and acquires voice data from the caller. The input at this time is a real-time voice signal, and the output is the acquired voice data. The voice response system uses a microphone to capture the voice signal.

[0645] Step 2:

[0646] The server converts the acquired audio data into text data using a speech recognition API. The input is audio data, and the output is the corresponding text data. A speech recognition service such as Google Cloud Speech-to-Text is used for this conversion.

[0647] Step 3:

[0648] The server analyzes text data using a natural language processing library and summarizes the sender's requirements. The input is text data, and the output is the summarized content. A natural language processing tool like spaCy extracts keywords and context through the analysis.

[0649] Step 4:

[0650] The server compares the caller's identification information with a database and calculates the confidence score. The input is the caller's identification information, and the output is the caller's confidence score. The calculation is based on past communication history and evaluations.

[0651] Step 5:

[0652] The server decides whether to allow or deny a call based on a confidence score. The input is the confidence score, and the output is whether the call is allowed or denied. If the confidence score falls below a certain threshold, the call may be denied.

[0653] Step 6:

[0654] The server uses the emotion recognition engine built into the terminal to analyze the voice tone and speech rate during communication in real time. The input is text data and voice characteristics, and the output is the emotion analysis result. A generative AI model detects emotions and determines specific emotional states such as "joy" or "anger."

[0655] Step 7:

[0656] The device issues an alert when the user experiences stress. The input is the result of sentiment analysis, and the output is an alert to the user. The alert is delivered to the user as a visual or audio notification.

[0657] Step 8:

[0658] An interface is provided for the user to choose a course of action. The input is the user's selection, and the output is an action such as continuing or ending the call. A prompt such as "Do you want to continue the call?" is displayed to the user.

[0659] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0660] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0661] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0662] [Fourth Embodiment]

[0663] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0664] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0665] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0666] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0667] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0668] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0669] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0670] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0671] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0672] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0673] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0674] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0675] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0676] This invention provides a system that enables users, including the elderly and those with dementia, to engage in safe and reliable communication. Specific embodiments of the system are described below.

[0677] In this invention, the server acts as a communication device, receiving all incoming calls first. Upon receiving an incoming call, the server activates an AI-powered voice response system and automatically sends a message to the caller saying, "Please tell us your name and purpose of your call." The caller's voice response is converted into text data in real time by the terminal.

[0678] The terminal summarizes this text data using natural language processing technology and stores it in a structured format. Next, the server matches the caller's identification information (e.g., phone number) against a dedicated database. This database stores information on past communication history and the trustworthiness of each caller, and a trust score for the caller is calculated based on this information.

[0679] The confidence score serves as a criterion for the server to decide whether to allow a call. If the confidence score exceeds a certain level, the server will make the user's device ring to inform the user of the intention to call. This allows the user to receive only calls from trusted callers. On the other hand, if the confidence score is low, the server will automatically terminate the call or switch to voicemail mode.

[0680] Even after the user answers a call, the device continues to monitor the call content in real time. If any unusual patterns or content that does not match the initial summary is detected during the call, the device can alert the user. For example, if an inappropriate request suddenly appears, the device will automatically sound an alert. This feature allows users to make calls with greater peace of mind.

[0681] A concrete example is a case where a server receives an incoming call, and the caller, claiming to be a "financial institution representative," attempts to verify account information. The server checks the caller's number in its database and calculates its reliability by referring to past records. If the number matches one suspected of being a fraudulent call, the server rejects the call and does not notify the user. In this way, the present invention prevents fraudulent communications and protects users securely.

[0682] The following describes the processing flow.

[0683] Step 1:

[0684] When the server detects an incoming call, it activates the voice response system. It automatically sends a message to the caller saying, "Please tell me your name and how can I help you?"

[0685] Step 2:

[0686] When the caller responds to a message, the device converts the audio data into text data in real time. Speech recognition technology is used to convert the caller's speech into digital text information.

[0687] Step 3:

[0688] The text data acquired by the terminal is fed into a natural language processing engine for summarization. This generates a concise summary of the caller's name and the purpose of the message.

[0689] Step 4:

[0690] The server checks the caller's phone number against a database and calculates a confidence score based on past communication history and ratings. It utilizes information from reliable data sources to quantify the caller's trustworthiness.

[0691] Step 5:

[0692] The server evaluates the trust score and decides whether to ring the user's device. If the trust score exceeds the threshold, the device rings to allow the user to make a call. If the trust score is low, the call is automatically rejected or switched to voicemail mode.

[0693] Step 6:

[0694] When a user starts a call, the device monitors the call content in real time. It transcribes the audio data during the call and verifies whether it matches the initial summary.

[0695] Step 7:

[0696] If unusual content or inconsistencies with the initial summary are detected during a call, the device will alert the user. If necessary, it will automatically terminate the call to protect the user.

[0697] By following the steps outlined above, this system can provide users with a safe and reliable calling environment.

[0698] (Example 1)

[0699] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0700] The goal is to provide a system that allows users, including the elderly and those with dementia, to communicate safely and securely when they receive fraudulent calls or unauthorized communications. Conventional communication methods make it difficult to accurately determine the caller's intentions, necessitating appropriate security measures. This system aims to prevent confusion and problems caused by incoming calls from unreliable callers.

[0701] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0702] In this invention, the server includes means for activating a voice response function using a generative model when a communication device receives an incoming call, presenting a prompt message to the caller to obtain voice information, converting the voice information into text and summarizing it using a natural language processing function, and comparing the caller's identification number with an information base to calculate a reliability rating. As a result, users can receive only communications from trusted callers, eliminate abnormal communications in advance, and maintain a safe and secure communication environment.

[0703] A "communication device" is a device that can send and receive voice and data, and has the function of starting the next processing step upon receiving an incoming call.

[0704] A "generative model" refers to an algorithm that operates using artificial intelligence, understands human language, and generates responses, and is used to drive voice response functions.

[0705] A "prompt message" is a message designed to prompt the caller for information or a response, and refers to questions or instructions automatically generated by voice response functions.

[0706] "Audio information" refers to audio data emitted by a caller, which is converted into text through speech recognition processing.

[0707] "Natural language processing" refers to the technology that allows computers to understand, interpret, and generate human language, and is used to summarize text information.

[0708] An "identification value" is numerical data used to identify the sender and verify the origin of the message.

[0709] An "information base" refers to a data storage system that accumulates past communication history and sender reliability information, and is used as a criterion for reliability evaluation.

[0710] "Trust rating" is a numerical representation of the trustworthiness of a sender, calculated based on their past history and evaluations.

[0711] "Call information" refers to the audio and text data generated during a call, and is the content of the communication that is subject to analysis.

[0712] This invention provides a system that enables users, including the elderly and those with dementia, to communicate safely and reliably. The system is comprised of a combination of voice response functionality, speech recognition technology, natural language processing technology, identification information matching, reliability evaluation, and call monitoring.

[0713] The server operates as a communication device and receives incoming calls from callers. Upon receiving a call, the server uses a generative AI model to activate its voice response function. This function generates a prompt message and presents it to the caller. An example of a prompt message used is, "Please tell us your name and purpose of your call." This prompt message serves to encourage the caller to provide a clear response.

[0714] The device utilizes speech recognition technology to convert the caller's voice information into text in real time. Specifically, general-purpose speech recognition software is used. The converted text data is summarized using natural language processing techniques. This allows the caller's requirements and intentions to be clarified in a short amount of time.

[0715] The server compares the caller's identification number with an information base and calculates a trust rating. This information base stores past communication history and ratings of the caller's trustworthiness. Based on this trust rating, the server decides whether to allow or deny the call.

[0716] Even if the user answers, the terminal continues to analyze the call information, and if it detects suspicious patterns or content that does not match the initial summary, it will warn the user. In this way, users can communicate securely and reliably.

[0717] A concrete example is a case where the caller impersonates a financial institution employee and attempts to verify account information. In this case, the server checks the caller's identification number and rejects the call if it is suspected to be fraudulent. At the same time, the user receives a warning about the suspicious content. This system prevents fraudulent communication and ensures the safety of users.

[0718] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0719] Step 1:

[0720] The server receives the incoming call. The input is an audio signal along with the caller's identifier. This incoming signal invokes the voice response function. The output is a preparation to issue a prompt to the caller using a generative AI model. This prompt is "Please tell us your name and purpose of call."

[0721] Step 2:

[0722] The device converts the caller's voice response into text data in real time. It receives the voice signal as input and digitizes it as text data using speech recognition technology. The output is text data in string format. Specifically, the content spoken by the caller is accurately transcribed into text.

[0723] Step 3:

[0724] The terminal summarizes this text data using natural language processing techniques. The input is the text data obtained in step 2. A natural language processing algorithm is applied to this data to extract important information and generate a simplified summary. The output is a summarized text, making the sender's requirements easier to understand.

[0725] Step 4:

[0726] The server compares the caller's identification number against a dedicated information base. The input is the caller identification data received in step 1. By comparing it with past communication history stored in the information base, the server calculates a trust rating for the caller. The output is a trust score. Based on this score, the server is ready to decide whether to allow or deny the call.

[0727] Step 5:

[0728] The server decides how to handle the call based on the trust score. The input is the trust score calculated in step 4. If the trust score is above a certain level, the server rings the user's device and allows the call. Conversely, if the trust score is low, the server rejects the call or switches to voicemail mode. The output is whether the call was allowed or terminated.

[0729] Step 6:

[0730] When a user receives a call, the device monitors the call content. It converts the audio data into text in real time and verifies that it matches the initial summary. The input is the audio data generated during the call. As output, it warns the user if any unusual patterns are detected. Specifically, an alert will sound if a sudden, inappropriate request occurs.

[0731] (Application Example 1)

[0732] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0733] The communication risks that users, including the elderly and those with dementia, face on a daily basis can cause serious problems, particularly through fraudulent attempts such as scams. This invention aims to prevent such fraudulent communications and provide a safe and reliable calling experience. In particular, it is necessary to provide a method for detecting fraudulent calls in real time and preventing harm to users.

[0734] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0735] In this invention, the server includes means for activating a voice response system when a communication device receives an incoming call and acquiring voice data from the caller; means for converting the voice data into text data and summarizing it using natural language processing; means for comparing the caller's identification information with a database and calculating the reliability level; and means for re-evaluating the reliability level in real time and allowing only secure calls. This makes it possible to detect fraudulent communications as suspicious patterns and issue warnings to users.

[0736] "Communication equipment" refers to devices and means for receiving incoming calls, and plays a central role in acquiring and processing voice data.

[0737] A "voice response system" is a program or function that automatically provides a voice message to the caller upon receiving a call and collects necessary information.

[0738] "Audio data" refers to audio information obtained from the caller, and is subject to analysis and text conversion.

[0739] "Text data" refers to the written information that remains after audio data has been converted, and is a data format used for natural language processing and summarization.

[0740] "Natural language processing" refers to the technology of analyzing and processing human language using computers, and is a technical means of summarizing text data.

[0741] "Caller identification information" refers to data used to identify a caller, such as a phone number or other unique information.

[0742] A "database" is a collection of data that stores past communication history and evaluations, and is used as reference information for calculating reliability.

[0743] "Trustworthiness" is an indicator used to evaluate the sender's communication behavior and demonstrate the security and legitimacy of the communication.

[0744] "Real-time" is a term that indicates that information acquisition and processing occur virtually instantaneously.

[0745] "Anomaly" refers to behavior that deviates from normal communication patterns or suspicious points that suggest a risk of fraud.

[0746] This invention relates to a system for enabling safe and reliable communication for users, including the elderly and those with dementia. A server functions as the central component of this system and is implemented as follows:

[0747] When the server receives an incoming call, it automatically provides a message to the caller using a voice response system and collects the necessary audio data. This audio data is converted to text data in real time on the terminal. Software such as TensorFlow or natural language processing libraries (e.g., spaCy) are used for the audio-to-text conversion.

[0748] Furthermore, the server matches the caller's identification information against a database and refers to past communication history in the database to calculate the reliability score. This reliability score is updated in real time, enhancing the security of calls.

[0749] The user's device has a feature that only notifies the user of incoming calls if the call's trustworthiness exceeds a certain threshold. This ensures that the user only receives calls from trusted sources. If a suspicious pattern is detected, the device will alert the user. This strengthens protection, especially against potentially fraudulent calls.

[0750] For example, if a user receives a call claiming to be from the city hall with an "important announcement," the server will automatically block the call if the caller's trustworthiness is low. This also helps the user prevent becoming a victim of fraud. To standardize this response process, the following is used as an example of a prompt:

[0751] "Input: Convert the recorded audio of a phone call from the city hall into text and analyze it for any suspicious patterns."

[0752] "Output: Confidence score and warning message for suspicious patterns"

[0753] Thus, the system according to the present invention aims to provide a safe and smooth communication experience for the elderly and dementia patients.

[0754] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0755] Step 1:

[0756] When the server receives an incoming call, it activates the voice response system and automatically sends a message to the caller. The input is an incoming call, and the output is a voice message to the caller. This process captures the caller's voice data.

[0757] Step 2:

[0758] The device converts acquired audio data into text data in real time. The input is the acquired audio data, and the output is the corresponding text data. Speech recognition technology is used here; for example, TensorFlow is used to convert audio to text.

[0759] Step 3:

[0760] The server applies natural language processing to text data to create a summary. The input is text data, and the output is summarized text. Here, a generative AI model is used to summarize the data, structuring the information.

[0761] Step 4:

[0762] The server compares the caller's identification information with a database and calculates the confidence score. The input is the caller's identification information and the database used for comparison, and the output is the confidence score. This confidence score is used to evaluate the caller's safety.

[0763] Step 5:

[0764] The server decides whether to allow or deny a call based on a confidence score. The input is the confidence score, and the output is whether the call is allowed or denied. Based on the set threshold, the call is only allowed if the caller is deemed trustworthy.

[0765] Step 6:

[0766] If a call is permitted, the user's device analyzes the call content in real time and monitors for any abnormalities. The input is the call content, and the output is whether or not there are any abnormalities. This monitoring allows the user to confirm that they are safe even while on a call.

[0767] Step 7:

[0768] If an anomaly is detected, the device will immediately issue a warning to the user. The input is the analyzed call content, and the output is a warning notification. Specifically, if a suspicious request is detected, the device will alert the user by displaying a warning sound or notification.

[0769] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0770] This invention is a system that combines an emotion engine for recognizing user emotions with an incoming call management system in a communication device. Specific embodiments of this system are described below.

[0771] When the server detects an incoming call, it activates the voice response system and requests a voice response from the caller. The caller's response is received as voice data by the terminal and converted to text. This text data is summarized using natural language processing techniques, and the caller's name and request are analyzed.

[0772] Simultaneously, the server compares the caller's identification information with a database and calculates a confidence score based on past communication history and evaluations. Based on this confidence score, the server decides whether to allow the user to make a call.

[0773] Furthermore, during a call, the device's built-in emotion engine analyzes the user's voice tone, speaking speed, and word choice to recognize emotions in real time. The emotion engine detects basic emotions such as joy, anger, surprise, sadness, and anxiety, and supports decision-making based on the communication situation. Based on this information, the server can decide whether to continue the call, interrupt it, or issue additional warnings.

[0774] As a concrete example, consider a scenario where a user answers a call and the caller begins speaking in an aggressive tone. The emotion engine recognizes this as "anger," and if the user starts to feel stressed, the device can emit a warning sound or display a message on the screen asking, "Do you want to continue the call?" There is also an option to automatically terminate the call if the emotional escalation occurs.

[0775] Thus, by combining emotion analysis using an emotion engine and realizing even more advanced call management, the present invention can provide users with a safer and more comfortable calling environment.

[0776] The following describes the processing flow.

[0777] Step 1:

[0778] When the server detects an incoming call, it activates the voice response system. The server automatically sends a voice message to the caller saying, "Please tell me your name and how can I help you?"

[0779] Step 2:

[0780] When the caller answers, the terminal acquires the audio data and converts it into text data in real time. Speech recognition technology is used in this process.

[0781] Step 3:

[0782] The terminal feeds the text data into a natural language processing engine to generate a summary. This summarizes the caller's name and the main points of the request.

[0783] Step 4:

[0784] The server checks the caller's phone number against a database. The database contains records of past communication history and ratings, and the server calculates a confidence score based on this information.

[0785] Step 5:

[0786] The server evaluates the confidence score and decides whether to ring the user's device. If the score exceeds a certain threshold, the device will ring to inform the user of the purpose of the call. Otherwise, the call will be automatically rejected or switched to voicemail mode.

[0787] Step 6:

[0788] When a user initiates a call, the device activates its emotion engine. The emotion engine analyzes the user's voice tone and speaking speed in real time and evaluates their emotional state.

[0789] Step 7:

[0790] When the emotion engine detects stressful states such as "anger" or "anxiety," the device issues a warning to the user. Specifically, it sounds a warning and displays a warning message on the screen.

[0791] Step 8:

[0792] In response to the device's warning, the user can choose to continue or end the call. The system can also automatically terminate the call if there are signs of emotional escalation.

[0793] Through this series of processing steps, the system provides users with a safe and comfortable calling environment.

[0794] (Example 2)

[0795] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0796] Modern communication devices are required to appropriately assess caller information and emotional state during calls to provide a safe and comfortable calling experience. However, conventional systems have not adequately verified caller information or analyzed emotions during calls, potentially leading to unpleasant experiences for users. Therefore, the challenge is to provide technology that enables appropriate call management based on caller reliability and emotional changes during calls.

[0797] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0798] In this invention, the server includes means for activating a voice response function when a communication device detects an incoming call and obtaining voice information from the caller; means for converting the voice information into text information and summarizing it using information processing technology; means for comparing the caller's identification information with a recording device and calculating the reliability level; means for analyzing emotions in real time during a call using an information analysis engine and providing warnings according to those emotions; and means for automatically terminating a call when emotions exceed a certain threshold. This enables users to have a safe and comfortable calling environment.

[0799] "Communication equipment" refers to devices used to send and receive voice and data, and includes telephones, smartphones, and network devices.

[0800] "Voice response function" refers to a system in which a communication device automatically receives voice messages and generates the necessary response.

[0801] "Audio information" refers to data transmitted through human voices, and includes phone calls and recorded audio.

[0802] "Text information" refers to information expressed as a string of characters, and includes data obtained by converting audio information into text.

[0803] "Information processing technology" refers to computer technologies used to analyze, transform, and utilize data, and includes natural language processing, among others.

[0804] "Identification information" refers to data used to identify a specific individual or sender, and includes names and ID numbers.

[0805] A "recording device" refers to a device for storing information, and this includes databases and storage systems.

[0806] "Trustworthiness" is a value that indicates how trustworthy a particular piece of information or its source is, and is calculated based on past history and evaluations.

[0807] An "information analysis engine" refers to software or algorithms used to analyze data and extract meaning and patterns.

[0808] "Emotions" refer to the psychological state exhibited by the caller and receiver during a phone call, and include feelings such as joy, anger, surprise, sadness, and anxiety.

[0809] A "standard" refers to a benchmark or reference point used when making a particular action or judgment.

[0810] This invention combines an emotion recognition engine with an incoming call management system for communication devices, providing a system that enables users to have safer and more comfortable phone calls. This system mainly consists of a server and terminals, each performing its own specific role.

[0811] The server's processing begins when the communication device detects an incoming call; it activates the voice response function to obtain voice information from the caller. The voice response system uses any speech synthesis technology to request the caller to provide their name and purpose of the call. The acquired voice information is then converted into text on the server. For speech recognition, commonly used technologies such as the Google Speech-to-Text API or open-source speech recognition libraries can be utilized.

[0812] The server analyzes the converted text information using information processing technology and calculates a confidence score by comparing the caller's identification information with the recording device. This can be done by applying natural language processing technology and incorporating libraries such as Apache OpenNLP and Stanford NLP. The confidence score thus calculated is used as a criterion for deciding whether to allow or deny the call.

[0813] During a call, the device uses an information analysis engine to analyze the user's emotions in real time. This emotion engine employs technologies such as Affectiva and Emotion AI to identify emotions from changes in the user's voice tone and speaking style. This emotion data is sent to a server and contributes to issuing warnings based on the communication status and determining when to end the call.

[0814] As a concrete example, consider a scenario where the caller's voice becomes aggressive during a call. When the emotion engine recognizes "anger," the device emits a warning sound and displays the option "Do you wish to continue the call?" on the screen. Furthermore, a feature is provided that allows the user to automatically terminate the call if such emotions escalate.

[0815] A concrete example of a prompt for a generative AI model could be: "What is the most effective way to warn a user when they are receiving a call from someone speaking in an aggressive tone?"

[0816] This system allows users to enjoy a more comfortable and secure calling environment based on the caller's trustworthiness and emotional state during the call.

[0817] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0818] Step 1:

[0819] The server detects an incoming call from the communication device and activates the voice response function. The server receives the incoming signal as input and issues a command to activate the voice response system. This process prepares the server to begin providing a voice message to the caller.

[0820] Step 2:

[0821] The server obtains voice information from the caller. The voice response system receives the caller's response in voice format and sends it to the server. The input is the caller's raw voice data, which is stored on the server for later processing.

[0822] Step 3:

[0823] The server converts the audio information into text information. The server uses speech recognition software to process the acquired audio data and generate text data. This conversion makes the audio content available as text information.

[0824] Step 4:

[0825] The server analyzes and summarizes text information. Utilizing natural language processing technology, it processes the input text information to extract important elements such as the caller's name and purpose. The output is summarized information, which can be used for call management.

[0826] Step 5:

[0827] The server compares the caller's identification information with the recording device and calculates the confidence level. The server takes caller identification data as input and evaluates the confidence level by comparing it with a database. The confidence level calculated based on past communication history is stored as output within the server.

[0828] Step 6:

[0829] The server decides whether to allow or deny the call based on its trustworthiness. Based on the acquired trustworthiness, the server makes a decision: if it allows the call, it continues the call; if it denies it, it terminates the call. The output is determined as either the allowed or denied status of the call.

[0830] Step 7:

[0831] The device analyzes the user's emotions in real time during a call. The device uses a data analysis engine to identify emotions based on the user's recorded voice. The recognized emotion information is sent to a server and used to inform communication decisions.

[0832] Step 8:

[0833] The server provides users with emotionally-sensitive warnings. Based on emotional information, the server displays warning sounds or messages to the user via the terminal if necessary, ensuring the safety of the call. It receives emotional information as input and utilizes warning methods as output.

[0834] Step 9:

[0835] The server automatically terminates the call when emotions exceed a certain threshold. The server compares emotional information with the set threshold and, if the threshold is exceeded, automatically outputs a command to end the call. This action helps users avoid unnecessary stress and unpleasant situations.

[0836] (Application Example 2)

[0837] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0838] There is a need to improve the safety and comfort of calls using communication devices. In particular, there is a lack of means to recognize the caller's emotions and stress levels in real time, which presents a challenge in responding appropriately to situations where users feel anxious or stressed. In this situation, an effective system is needed to mitigate the risk of users encountering unforeseen circumstances.

[0839] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0840] In this invention, the server includes means for activating a voice response system when a communication device receives an incoming call and acquiring voice data from the caller, means for analyzing voice tone and speaking speed to recognize emotions in real time, and means for issuing an alert when the user feels stressed. This makes it possible to analyze the caller's emotional state and provide appropriate feedback to the user.

[0841] A "communication device" is a device that transmits and receives voice data, enabling communication between users.

[0842] A "voice response system" is a system that automatically activates when a call is received and receives voice data from the caller.

[0843] "Audio data" refers to data that records the speaker's utterance as a digital signal.

[0844] "Natural language processing" is an information processing technology that analyzes text data and derives its meaning and summary.

[0845] "Caller identification information" refers to information used to identify a caller, and includes information such as a phone number and name.

[0846] "Trustworthiness" is an indicator of whether a sender is trustworthy, and it is a score based on past communication history and evaluations.

[0847] "Voice tone" refers to the characteristics of pitch and intensity of a voice that reflect the speaker's psychological state and emotions.

[0848] "Speech rate" is an indicator that shows the number of words a speaker utters within a certain period of time, and is a standard for measuring the pace of communication.

[0849] "Means of recognizing emotions in real time" refers to technologies and methods for instantly determining the emotional state of a speaker while they are speaking.

[0850] "Means of issuing alerts" refers to notification functions used to warn or alert users.

[0851] To realize this invention, it is necessary to install a dedicated application on communication devices such as smartphones and servers. The main functions of the system are voice data processing and emotion recognition, and it operates as follows.

[0852] First, when a communication device receives an incoming call, the server activates the voice response system and retrieves the caller's voice data. This data is converted into text data using a speech recognition API (e.g., Google Cloud Speech-to-Text). Next, the text is summarized using a natural language processing library (e.g., spaCy), and the caller's identification information and confidence level are calculated through database matching.

[0853] The server further analyzes the received voice tone and speaking speed in real time using an emotion recognition engine built into the device. This analysis detects the caller's emotions, such as joy or anger. If the user feels stressed, the device issues an alert and presents the user with options such as "Do you want to continue the call?". It can also automatically terminate the call if the emotions escalate.

[0854] For example, if the system detects unsettling emotions during a call, it will automatically trigger a prompt in real time to "monitor the caller's emotions and notify the user if their tone is unsettling." This mechanism protects users from unexpected and unpleasant interactions.

[0855] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0856] Step 1:

[0857] The server activates the voice response system when the communication device receives an incoming call and acquires voice data from the caller. The input at this time is a real-time voice signal, and the output is the acquired voice data. The voice response system uses a microphone to capture the voice signal.

[0858] Step 2:

[0859] The server converts the acquired audio data into text data using a speech recognition API. The input is audio data, and the output is the corresponding text data. A speech recognition service such as Google Cloud Speech-to-Text is used for this conversion.

[0860] Step 3:

[0861] The server analyzes text data using a natural language processing library and summarizes the sender's requirements. The input is text data, and the output is the summarized content. A natural language processing tool like spaCy extracts keywords and context through the analysis.

[0862] Step 4:

[0863] The server compares the caller's identification information with a database and calculates the confidence score. The input is the caller's identification information, and the output is the caller's confidence score. The calculation is based on past communication history and evaluations.

[0864] Step 5:

[0865] The server decides whether to allow or deny a call based on a confidence score. The input is the confidence score, and the output is whether the call is allowed or denied. If the confidence score falls below a certain threshold, the call may be denied.

[0866] Step 6:

[0867] The server uses the emotion recognition engine built into the terminal to analyze the voice tone and speech rate during communication in real time. The input is text data and voice characteristics, and the output is the emotion analysis result. A generative AI model detects emotions and determines specific emotional states such as "joy" or "anger."

[0868] Step 7:

[0869] The device issues an alert when the user experiences stress. The input is the result of sentiment analysis, and the output is an alert to the user. The alert is delivered to the user as a visual or audio notification.

[0870] Step 8:

[0871] An interface is provided for the user to choose a course of action. The input is the user's selection, and the output is an action such as continuing or ending the call. A prompt such as "Do you want to continue the call?" is displayed to the user.

[0872] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0873] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0874] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0875] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0876] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0877] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0878] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0879] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0880] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0881] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0882] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0883] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0884] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0885] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0886] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0887] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0888] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0889] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0890] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0891] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0892] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0893] The following is further disclosed regarding the embodiments described above.

[0894] (Claim 1)

[0895] A means for activating a voice response system when a communication device receives an incoming call and obtaining voice data from the caller,

[0896] A means of converting audio data into text data and summarizing it using natural language processing,

[0897] A means of comparing the sender's identification information with a database and calculating the reliability,

[0898] A means of making decisions to allow or deny a call based on trustworthiness,

[0899] A means of analyzing call content and issuing a warning if an anomaly is detected,

[0900] A system that includes this.

[0901] (Claim 2)

[0902] The system according to claim 1, characterized in that it takes into account past records and evaluations when calculating the reliability level.

[0903] (Claim 3)

[0904] The system according to claim 1, characterized in that it transcribes audio data during a call into text in real time and verifies that it matches the initial summary content.

[0905] "Example 1"

[0906] (Claim 1)

[0907] A means for activating a voice response function using a generative model when a communication device receives an incoming call, presenting a prompt message to the caller, and obtaining voice information,

[0908] A means of converting audio information into text and summarizing it using natural language processing functions,

[0909] A means of calculating a reliability rating by comparing the sender's identification number with an information base,

[0910] A means of making a decision to allow or deny a call based on a trust assessment,

[0911] A means of analyzing call information and issuing a warning if an anomaly is detected,

[0912] A system that includes this.

[0913] (Claim 2)

[0914] The system according to claim 1, characterized in that it takes into account past history and evaluations when calculating the reliability rating.

[0915] (Claim 3)

[0916] The system according to claim 1, characterized in that it transcribes audio information during a call into text in real time and verifies that it matches the initial summary information.

[0917] "Application Example 1"

[0918] (Claim 1)

[0919] A means for activating a voice response system when a communication device receives an incoming call and obtaining voice data from the caller,

[0920] A means of converting audio data into text data and summarizing it using natural language processing,

[0921] A means of comparing the sender's identification information with a database and calculating the reliability,

[0922] A means of making decisions to allow or deny a call based on trustworthiness,

[0923] A means of analyzing call content and issuing a warning if an anomaly is detected,

[0924] A means to reassess reliability in real time and allow only secure calls,

[0925] A system that includes this.

[0926] (Claim 2)

[0927] The system according to claim 1, characterized in that it uses past records and evaluations as criteria for calculating reliability and detects communications that are suspected of being fraudulent.

[0928] (Claim 3)

[0929] The system according to claim 1, characterized in that it transcribes audio data during a call into text in real time, verifies that it matches the initial summary, and continuously monitors reliability to ensure a secure call.

[0930] "Example 2 of combining an emotion engine"

[0931] (Claim 1)

[0932] A means of activating the voice response function when a communication device detects an incoming call and obtaining voice information from the caller,

[0933] A means of converting audio information into text information and summarizing it using information processing technology,

[0934] A means for comparing the caller's identification information with a recording device and calculating the reliability,

[0935] A means of making decisions to allow or deny a call based on trustworthiness,

[0936] A means of analyzing emotions in real time during a call using an information analysis engine and providing warnings tailored to those emotions,

[0937] A means to automatically terminate a call when emotions exceed a certain threshold,

[0938] A system that includes this.

[0939] (Claim 2)

[0940] The system according to claim 1, characterized in that it takes into account past communication history and evaluations when calculating reliability.

[0941] (Claim 3)

[0942] The system according to claim 1, characterized in that it transcribes audio information during a call into text in real time and verifies its consistency with the initial summary content.

[0943] "Application example 2 when combining with an emotional engine"

[0944] (Claim 1)

[0945] A means for activating a voice response system when a communication device receives an incoming call and obtaining voice data from the caller,

[0946] A means of converting audio data into text data and summarizing it using natural language processing,

[0947] A means of comparing the sender's identification information with a database and calculating the reliability,

[0948] A means of making decisions to allow or deny a call based on trustworthiness,

[0949] A means of analyzing call content and issuing a warning if an anomaly is detected,

[0950] A method for analyzing voice tone and speech rate to recognize emotions in real time,

[0951] A means of issuing an alert when a user feels stressed,

[0952] A system that includes this.

[0953] (Claim 2)

[0954] The system according to claim 1, characterized in that it takes into account past records and evaluations when calculating the reliability level.

[0955] (Claim 3)

[0956] The system according to claim 1, characterized in that it transcribes audio data during a call into text in real time and verifies that it matches the initial summary content. [Explanation of Symbols]

[0957] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for activating a voice response system when a communication device receives an incoming call and obtaining voice data from the caller, A means of converting audio data into text data and summarizing it using natural language processing, A means of comparing the sender's identification information with a database and calculating the reliability, A means of making decisions to allow or deny a call based on trustworthiness, A means of analyzing call content and issuing a warning if an anomaly is detected, A system that includes this.

2. The system according to claim 1, characterized in that it takes into account past records and evaluations when calculating the reliability level.

3. The system according to claim 1, characterized in that it transcribes audio data during a call into text in real time and verifies that it matches the initial summary content.

Images