System

Abstract

A system is provided.SOLUTION: The system includes a means for acquiring voice data in real time, a means for converting the acquired voice data into text data, a means for analyzing a possibility of fraud from the converted text data, a means for issuing a warning when there is a possibility of fraud, and a means for notifying a family member of the possibility of fraud.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, the number of victims of telephone fraud targeting the elderly has been increasing, and the amount of damage caused has become a major social problem. Such fraud victims steal the elderly's valuable assets and pensions, causing great damage not only to the elderly themselves but also to their families. In particular, the elderly are less vigilant against sophisticated fraud methods and are easily deceived. Therefore, effective means to prevent fraud damage are needed. Conventional technology focuses on educating people to be vigilant against fraud methods, and there is an issue of a lack of technology that can respond immediately to actual fraud situations. [Means for solving the problem]

[0005] The present invention provides a system that acquires voice data in real time, converts the voice data into text data, and analyzes the converted text data for possible fraud. It also proposes a system that includes a means for issuing a warning when there is a possibility of fraud, and a means for notifying relatives of the possibility of fraud. The system of the present invention is characterized by including the following components:

[0006] A means of capturing audio data in real time

[0007] A means for converting acquired voice data into text data

[0008] A method for analyzing the possibility of fraud from converted text data

[0009] A means of alerting you to potential fraud

[0010] How to notify relatives of potential fraud

[0011] This will enable real-time responses when seniors receive fraudulent phone calls, preventing them from becoming victims of fraud. In addition, because this system runs on local devices, it provides highly accurate fraud detection capabilities while reducing the risk of personal information leaks.

[0012] "Voice data" means data that is a digital representation of the human voice, such as a telephone call or a recorded conversation.

[0013] "Real-time" refers to the ability to process and respond immediately, with minimal delay or latency.

[0014] "Means for obtaining" refers to a device or method for collecting or obtaining specific information or data.

[0015] "Text data" refers to data that is stored, displayed, or processed in text format, such as characters or sentences.

[0016] A "transforming means" is a device or method for changing data or information from one form to another.

[0017] "Means of analysis" are devices and methods for analyzing data and information and understanding its content and trends.

[0018] "Potential fraud" means that a particular action or statement may constitute fraud.

[0019] A "means for issuing warnings" is a device or method that provides a user with a notice of a hazard or caution.

[0020] "Kinship" refers to family members or relatives related by blood or marriage.

[0021] A "notification means" is a device or method for conveying specific information or messages to other people or devices.

[0022] A "system" refers to a configuration in which multiple devices or methods work together to achieve a specific function or purpose.

[0023] A "local terminal" is a device such as a computer or smartphone that can be directly operated by an individual user. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0025] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0026] First, the terms used in the following description will be explained.

[0027] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0028] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0029] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0030] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0031] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0032] [First embodiment]

[0033] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0034] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0035] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0036] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0037] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0038] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0039] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0040] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0041] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0042] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0043] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0044] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0045] This invention is a system for analyzing voice data in real time and providing warnings and notifications, designed to prevent elderly people from becoming victims of telephone fraud. The system of the present invention has the following configuration in particular.

[0046] System configuration

[0047] 1. A means of acquiring audio data in real time

[0048] Device: The user's device is equipped with real-time audio capture capabilities, which allows for instantaneous capture of phone conversations.

[0049] 2. A means of converting the acquired voice data into text data

[0050] Terminal (local AI model): Equipped with a speech recognition engine, it converts call content into text data in real time, allowing voice data to be handled as text information.

[0051] 3. A method for analyzing the possibility of fraud from the converted text data

[0052] Device (Local AI Model): Natural Language Processing (NLP) techniques are used to analyze the converted text data and detect fraudulent words and phrases. The local AI model determines the likelihood of fraud based on known fraud techniques.

[0053] 4. A way to flag potential fraud

[0054] Device: If a potential scam is detected, the device will immediately issue an audio warning to the user, including specific instructions such as "This call may be fraudulent. Do not continue the call."

[0055] 5. How to notify relatives of potential fraud

[0056] Server: If a potential scam is detected, the user's relatives will be notified via email or SMS, with a message such as "The user has received a potentially fraudulent call. Please check."

[0057] Program processing

[0058] Initial Setup

[0059] User: The user installs the "Phone Fraud Prevention App" (hereinafter referred to as the "App") on their device. After installation, they register the contact information of their relatives on a simple setup screen.

[0060] Server: The server receives user registration information and performs authentication and configuration.

[0061] Real-time call analysis

[0062] User: When a user answers the phone, the call begins.

[0063] Terminal: When a call begins, the terminal begins capturing voice data in real time.

[0064] Device (local AI model): The speech recognition engine converts voice data into text in real time.

[0065] On-device (local AI model): Analyzes the converted text data and filters it to see if it contains malicious words or phrases.

[0066] On-device (local AI model): Uses advanced natural language processing capabilities to understand context and assess the likelihood of fraud.

[0067] Fraud detection and notification

[0068] Device (local AI model): Activate fraud alert flag if fraud is determined to be highly likely.

[0069] Device: If the fraud alert flag is activated, the user will hear a voice warning saying, "This call may be fraudulent. Do not continue."

[0070] Device: At the same time, your relatives will be notified by email or SMS of the possible fraud.

[0071] Ending a call and saving a log

[0072] User: The user ends the call.

[0073] Device: After the call is completed, the call details and fraud detection results are saved in a secure log file.

[0074] Device: Saved logs are stored in secure storage on the user's device and can be viewed by the user or their relatives if necessary.

[0075] Specific examples

[0076] One day, the user receives a call on their mobile phone from someone claiming to be from a "public institution." As soon as the call begins, the device begins capturing the call content in real time. The local AI model converts the voice data into text and analyzes the call content. If the call content contains keywords associated with fraud, such as "unpaid bills" or "transfers," the local AI model determines that it is likely a scam. The device then immediately issues a voice warning to the user and simultaneously sends a notification to their relatives. When the call ends, the suspected fraudulent call content and the fraud determination result are recorded in a log and securely stored. This series of processes significantly reduces the risk of elderly people falling victim to fraud.

[0077] The system can provide peace of mind to seniors and their families through real-time voice analysis and immediate response.

[0078] The processing flow will be explained below.

[0079] Step 1:

[0080] User: The user installs the "Phone Fraud Prevention App" on their device. After installation, the initial setup screen is displayed. The user enters the contact information of their relatives and completes the setup.

[0081] Step 2:

[0082] Server: The server receives the user registration information and performs an authentication process. If authentication is successful, the server stores the registration information in a secure database.

[0083] Step 3:

[0084] User: When a call comes in to the user's phone, the user answers the call.

[0085] Step 4:

[0086] Terminal: When a call starts, the terminal starts capturing audio data in real time, which is temporarily stored in the terminal's buffer memory.

[0087] Step 5:

[0088] Device (local AI model): The speech recognition engine converts voice data into text data in real time. This conversion process is performed on the device, and the converted text data is sent to the analysis module.

[0089] Step 6:

[0090] On-device (local AI model): The analysis module receives the converted text data and filters it to analyze potential fraud. If specific keywords or phrases are detected, the content is analyzed in detail.

[0091] Step 7:

[0092] On-device (local AI model): An advanced natural language processing (NLP) engine understands the context and assesses the likelihood of fraud from text data. If fraud is deemed likely, a fraud alert flag is raised.

[0093] Step 8:

[0094] Terminal: When the fraud alert flag is activated, the terminal will immediately issue an audio warning to the user, playing a specific warning message such as "This call may be fraudulent. Do not continue the call."

[0095] Step 9:

[0096] At the same time, the device will send emails and / or SMS to registered contacts of relatives informing them of the potential fraud, including a brief description of the alleged fraud.

[0097] Step 10:

[0098] User: The user receives a warning and ends the call. It is recommended that the call be ended immediately after the warning.

[0099] Step 11:

[0100] Device: After the call ends, the device saves the call details and fraud detection results in a secure log file that is stored in secure storage on the user's device.

[0101] Step 12:

[0102] Device: The user or a relative can review this log file if necessary. The log file contains a portion of the call (suspected fraudulent portion) and details of the fraud determination.

[0103] This series of processing steps results in a system that provides real-time protection for seniors against telephone fraud.

[0104] Example 1

[0105] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0106] As the elderly are increasingly vulnerable to telephone fraud, there is a need to provide a safe calling environment. Conventional countermeasures against telephone fraud are mostly reactive and do not provide real-time fraud prevention measures. This means that the risk of elderly people falling victim to fraud and suffering financial and psychological damage cannot be fully prevented.

[0107] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0108] In this invention, the server includes means for acquiring voice data in real time, means for converting the acquired voice data into text data, means for analyzing the possibility of fraud from the converted text data, means for issuing a warning when there is a possibility of fraud, means for notifying relatives of the possibility of fraud, means for saving data after the call ends, and means for storing the analyzed data in secure storage. This reduces the risk of elderly people falling victim to telephone fraud in real time and enables immediate contact with relatives, thereby providing a safe calling environment.

[0109] "Voice data" means information in analog or digital form that contains the human voice.

[0110] "Real-time" means that processing occurs immediately, without delay.

[0111] A "capturing means" is a method or device for capturing or collecting audio data.

[0112] "Text data" is voice data expressed as text information.

[0113] A "converting means" is a method or device for converting audio data into text data.

[0114] The "analyzing means" refers to a method or device for analyzing the converted text data to detect specific patterns or keywords.

[0115] "Probability of fraud" is the probability that a communication is deemed to contain fraud or fraudulent intent.

[0116] "Warning means" refers to a method or device for sending a sound or message to alert the user.

[0117] "Means of notification" refers to the method or device used to communicate information to relatives.

[0118] "Means for saving data after a call ends" refers to a method or device for saving the contents of the call and the analysis results.

[0119] "Secure storage" refers to storage environments and technologies that ensure data safety.

[0120] "Local terminal" refers to a device or equipment that is directly operated by a user.

[0121] A "local AI model" is an artificial intelligence technology that runs within a device and performs specific analyses and judgments.

[0122] A "means for detecting the start of a call" is a method or device for recognizing that a call has been received or started.

[0123] This invention is a system for preventing elderly people from becoming victims of telephone fraud, and performs real-time analysis of voice data, warnings, and notifications. The detailed configuration and operation of this system will be described below.

[0124] System configuration

[0125] 1. A means of acquiring audio data in real time

[0126] Device: The user's device is equipped with a built-in microphone to capture audio data in real time, in order to detect when a call starts and instantly capture the contents of the call.

[0127] 2. A means of converting the acquired voice data into text data

[0128] Terminal (local AI model): Equipped with a voice recognition engine (e.g., Google® Speech-to-Text or Microsoft® Azure® Cognitive Services), it converts the contents of the call into text data in real time, allowing the voice data to be handled as text information.

[0129] 3. A method for analyzing the possibility of fraud from the converted text data

[0130] Terminal (local AI model): The analysis engine uses natural language processing (NLP) technology to analyze the converted text data, detecting fraudulent words and phrases based on existing fraud techniques and determining the likelihood of fraud.

[0131] 4. A way to flag potential fraud

[0132] Device: If a call is deemed potentially fraudulent, the device will immediately issue an audio warning to the user, including messages such as "This call may be fraudulent. Do not continue the call."

[0133] 5. How to notify relatives of potential fraud

[0134] Server: If a potential scam is detected, the user's relatives will be notified via email and / or SMS, with a message such as "The user has received a potentially fraudulent call. Please check."

[0135] 6. How to save data after a call ends

[0136] On the device: After the call ends, the analyzed text data and the log file containing the fraud detection result are stored in a secure storage location. This storage process is performed in a secure manner so that it can be accessed by the user.

[0137] 7. A means of storing the analyzed data in secure storage

[0138] Device: Saved log files are stored in secure storage on the device and can be viewed by the user or their relatives if necessary.

[0139] Specific examples

[0140] One day, the user receives a call on their mobile phone from someone claiming to be from a public institution. As soon as the call begins, the device begins capturing the call content in real time. The local AI model converts the voice data into text and analyzes the text data. If the call content contains keywords associated with fraud, such as "unpaid bills" or "transfers," the local AI model determines that it is likely a scam. The device then immediately issues an audio warning to the user and simultaneously sends a notification to their relatives. After the call ends, the suspected fraudulent content of the call and the fraud determination result are recorded in a log file and securely stored. This series of processes significantly reduces the risk of elderly people falling victim to fraud.

[0141] Prompt Sentence Examples

[0142] "Please explain the system to be followed when an elderly person receives a fraudulent phone call."

[0143] The system can provide peace of mind to seniors and their families through real-time voice analysis and immediate response.

[0144] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0145] Step 1:

[0146] When a user answers a call, the call begins. The device detects the start of the call and begins capturing audio data using the built-in microphone. The input at this time is the audio signal during the call, and the output is audio data. Specifically, the built-in microphone of the device collects the telephone voice and converts it into digital audio data.

[0147] Step 2:

[0148] The device converts the captured voice data into text data in real time. This uses a local AI model speech recognition engine (e.g., Google Speech-to-Text or Microsoft Azure Cognitive Services). The input is voice data, and the output is text data. Specifically, the speech recognition engine analyzes voice patterns and converts them into corresponding text data.

[0149] Step 3:

[0150] The device analyzes the text data to assess the likelihood of fraud. A local AI model, the Natural Language Processing (NLP) engine, is responsible for this process. The input is the text data, and the output is a flag indicating the likelihood of fraud (a fraud alert flag). Specifically, the NLP engine filters keywords and phrases in the text to detect patterns associated with fraud.

[0151] Step 4:

[0152] When the fraud alert flag is activated, the terminal immediately issues an audio warning to the user. The input is the fraud alert flag and the output is the warning message. Specifically, the terminal plays the audio message "This call may be fraudulent. Do not continue the call."

[0153] Step 5:

[0154] When the fraud alert flag is activated, the server notifies the relative of a possible fraud. The input is a signal sent from the device to the server, and the output is a notification message to the relative. Specifically, the server sends an email or SMS to the relative's registered contacts saying, "A user has answered a suspected fraudulent call. Please check."

[0155] Step 6:

[0156] When the user ends the call, the device saves the call content and the fraud detection result in a secure log file. The input is a call end signal and text data containing the fraud detection result, and the output is a secure log file. Specifically, the device encrypts the data and saves it in secure storage.

[0157] Step 7:

[0158] The saved log file is stored in secure storage within the device. The input is a secure log file, and the output is saved in secure storage. Specifically, the device saves the log file according to a specified security protocol and allows the user or relatives to access it as needed.

[0159] (Application example 1)

[0160] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0161] There is a need to effectively reduce the risk of elderly people becoming victims of telephone fraud. However, it is difficult for elderly people to recognize the signs of fraud themselves, and they are unable to act quickly without appropriate warnings and notifications. To solve these problems, a system that processes voice data in real time and provides immediate warnings is required.

[0162] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0163] In this invention, the server includes a means for acquiring voice data in real time, a means for converting the acquired voice data into text data, a means for analyzing the possibility of fraud from the converted text data, a means for notifying relatives of the possibility of fraud if there is a possibility of fraud, and a means for issuing visual and audio warnings, thereby significantly reducing the risk of elderly people becoming victims of telephone fraud and enabling prompt action.

[0164] "Audio data" refers to recordings of voice during a call, etc., and is information that is subject to signal processing.

[0165] "Means of acquisition in real time" refers to technical means for collecting voice data immediately without delay.

[0166] "Text data" is voice data converted into character information, and is basic data for analysis.

[0167] "Means for converting" refers to the speech recognition technology used to convert voice data into text data.

[0168] "Means for analyzing" refers to algorithms and techniques used to process the captured and converted text data to assess the likelihood of fraud.

[0169] "Alert Measures" refers to the audio and visual interfaces used to notify the user of potential fraud.

[0170] "Means of notifying relatives of possible fraud" refers to notification techniques that use email, SMS, or other means to inform a user's relatives of possible fraud.

[0171] "Visual and audio warning means" refers to devices and technologies that display and transmit visual and audio warnings that are intuitively understandable to the user.

[0172] A "local terminal" is a device distributed to a user, and is an apparatus that has computational resources and storage for real-time processing.

[0173] "Local AI model" refers to an artificial intelligence model that runs within the device and incorporates learning algorithms to detect signs of fraud.

[0174] "Means for detecting the start of a call" refers to a technical means for recognizing that a telephone call has started and immediately starting to capture voice data.

[0175] This invention is a system for analyzing voice data in real time and providing warnings and notifications, designed to prevent elderly people from becoming victims of telephone fraud. The system of the present invention has the following configuration in particular.

[0176] System configuration

[0177] The system includes means for acquiring voice data in real time, means for converting the acquired voice data into text data, means for analyzing the converted text data for the possibility of fraud, means for issuing an alert when there is a possibility of fraud, means for notifying relatives of the possibility of fraud, and means for issuing visual and audio alerts, thereby enabling the user to immediately recognize the risk of fraud and the relatives to quickly respond to the crisis.

[0178] Program processing

[0179] Initial Setup

[0180] User: The user installs the "Phone Fraud Prevention App" on their device. After installation, they register the contact information of their relatives on a simple setup screen.

[0181] Server: The server receives user registration information and performs authentication and configuration.

[0182] Real-time call analysis

[0183] Terminal (smart glasses): When a call starts, the microphone built into the smart glasses captures voice data in real time.

[0184] Terminal: The acquired voice data is converted into text data using a voice recognition engine (e.g., Google API).

[0185] On-device (local AI model): Analyzes the converted text data to see if it contains fraudulent words or phrases. This uses the ScamPredictionModel.

[0186] Fraud detection and notification

[0187] Device (local AI model): Activate fraud alert flag if fraud is determined to be highly likely.

[0188] Device: If the fraud alert flag is activated, the user will be warned audibly, "This call may be fraudulent. Do not continue the call." A visual warning will also be displayed on the smart glasses display.

[0189] Server: At the same time, notify the relatives of the possible fraud via email or SMS (using the smtplib library).

[0190] Ending a call and saving a log

[0191] User: The user ends the call.

[0192] Device: After the call is completed, the call details and fraud detection results are saved in a secure log file.

[0193] On the device: The saved logs are stored in secure storage and can be viewed by the user or a relative if necessary.

[0194] Specific examples

[0195] One day, the user receives a call through their smart glasses from someone claiming to be from a government agency. As soon as the call begins, the smart glasses begin capturing the call content in real time. The local AI model converts the audio data into text and analyzes the call content. If the call content contains keywords associated with fraud, such as "unpaid bills" or "transfers," the local AI model determines that it is likely a scam. The smart glasses then immediately issue an audio and visual warning to the user and simultaneously send an emergency notification to their relatives. When the call ends, the suspected fraudulent call content and the fraud determination result are logged and securely stored.

[0196] Generative AI model prompt example

[0197] "Develop an AI model to identify potentially fraudulent phone call content. The model will take Japanese text data as input and determine whether it is likely to be fraudulent. Examples of phrases include: "unpaid money", "transfer". The output should be a Boolean value indicating the likelihood of fraud."

[0198] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0199] Step 1:

[0200] User initiates a call

[0201] Input: A user receives a call through their smart glasses.

[0202] How it works: The smart glasses' call start detection sensor recognizes the start of a call and starts collecting voice data.

[0203] Output: A call initiation signal is sent to the processing unit of the smart glasses.

[0204] Step 2:

[0205] Acquiring audio data

[0206] Input: Call initiation signal and call audio.

[0207] How it works: The smart glasses have a built-in microphone that captures audio data in real time, which is then converted into digital form and sent to a processing unit.

[0208] Output: Digital signal of audio data.

[0209] Step 3:

[0210] Converting audio data to text

[0211] Input: Digital signal of audio data.

[0212] How it works: Using a speech recognition engine (e.g. Google API) within the smart glasses, the voice data is converted into text data. During this conversion process, the voice signal is analyzed and converted into corresponding text information.

[0213] Output: Text data.

[0214] Step 4:

[0215] Text data analysis

[0216] Input: Text data.

[0217] How it works: A local AI model (ScamPredictionModel) is used to analyze text data. The analysis process detects keywords and context within the text data and assesses its likelihood of fraud.

[0218] Output: A flag indicating whether fraud is likely.

[0219] Step 5:

[0220] Fraud Alert Verification

[0221] Input: Possible fraud flag.

[0222] How it works: If the fraud alert flag is active, the system determines that fraud is likely.

[0223] Output: Alert signal.

[0224] Step 6:

[0225] Sending a warning to the user

[0226] Input: Alert signal.

[0227] How it works: The smart glasses provide users with visual and audio warnings, including a message on the display that reads "This call may be fraudulent. Do not continue the call," as well as an audio warning.

[0228] Output: A warning notice to the user.

[0229] Step 7:

[0230] Sending notifications to relatives

[0231] Input: Alert signal.

[0232] What it does: The server sends emergency notifications via email or SMS based on the contact information of the next of kin configured. It uses the smtplib library to send the notifications.

[0233] Output: Notification sent to relatives.

[0234] Step 8:

[0235] Call content and result logging

[0236] Input: call termination signal, call content, fraud detection result.

[0237] How it works: When a user ends a call, the device saves the call and the fraud detection result in a secure log file that is encrypted and stored in secure storage on the user's device.

[0238] Output: Secure log file.

[0239] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0240] This invention is a system that performs real-time analysis of voice data and provides warnings and notifications to protect elderly people from becoming victims of telephone fraud. In particular, by combining this system with an emotion engine that recognizes the user's emotions, this system enables flexible responses according to the user's emotional state.

[0241] System configuration

[0242] 1. A means of acquiring audio data in real time

[0243] Device: The user's device is equipped with the capability to capture phone conversations in real time.

[0244] 2. A means of converting the acquired voice data into text data

[0245] Device (local AI model): A speech recognition engine converts voice data into text data in real time. This conversion process is performed within the device.

[0246] 3. A method for analyzing the possibility of fraud from the converted text data

[0247] On-device (local AI model): The analytics module analyzes the converted text data and filters it to detect fraudulent words and phrases. It uses natural language processing (NLP) techniques to understand the context and assess the likelihood of fraud.

[0248] 4. A way to flag potential fraud

[0249] Terminal: If a call is deemed potentially fraudulent, the terminal will issue an audio warning to the user, such as a warning message saying, "This call may be fraudulent. Do not continue the call."

[0250] 5. How to notify relatives of potential fraud

[0251] Server: Notify the user's relatives of the potential scam via email or SMS. This notification may include a message such as "The user has received a potentially fraudulent call. Please check."

[0252] 6. Means of recognizing user emotions using an emotion engine

[0253] The device is equipped with an emotion engine that analyzes the user's emotional state based on voice data. The emotion engine analyzes the tone of voice and word choice to gauge the user's emotions (e.g., surprise, confusion, anger, etc.).

[0254] 7. A means for the emotion engine to tailor warning content based on the user's emotional state

[0255] Terminal: The emotion engine recognizes the user's emotional state and issues a warning message according to the user's emotion. For example, if the user is surprised, it will provide a gentle warning message such as "Please stay calm. This call may be a scam."

[0256] 8. A method for detailing notifications to relatives based on the user's emotional state

[0257] Device: If the user's emotional state exceeds a certain threshold, the notification to the next of kin will be detailed. For example, if the user is very distressed, the next of kin will receive a detailed message saying, "The user is very distressed. Please contact us immediately."

[0258] Program processing

[0259] Initial Setup

[0260] User: The user installs the "Phone Fraud Prevention App" on their device. After installation, they register the contact information of their relatives and the method of notification in case of an emergency on the settings screen.

[0261] Server: The server receives the user registration information, performs an authentication process, and, if successful, stores the registration information in a secure database.

[0262] Real-time call analysis and emotion recognition

[0263] User: When a call comes in to the user's phone, answer the call.

[0264] Terminal: When a call starts, the terminal starts capturing audio data in real time, which is temporarily stored in the terminal's buffer memory.

[0265] Device (local AI model): The speech recognition engine converts voice data into text data in real time.

[0266] Terminal (local AI model): The analysis module receives the converted text data and filters it to analyze possible fraud.

[0267] On-device (local AI model): A natural language processing engine understands the context and assesses the likelihood of fraud.

[0268] Terminal (emotion engine): Analyzes the user's emotional state based on voice data and recognizes emotions such as surprise, confusion, or anger.

[0269] Fraud detection and emotion-based warnings and notifications

[0270] Device (local AI model): Activate fraud alert flag if fraud is determined to be highly likely.

[0271] Terminal: When the fraud alert flag is activated, the emotion engine recognizes the user's emotional state and issues a warning message, such as "This call may be fraudulent. Do not continue the call."

[0272] On the device: At the same time, an email or SMS is sent to a relative notifying them of the possible scam. If the user's emotional state is very distressed, the notification will include a detailed message such as "The user is very distressed. Please contact us immediately."

[0273] Ending a call and saving a log

[0274] User: The user receives an alert and ends the call.

[0275] Device: After the call is completed, the call details and fraud detection results are saved in a secure log file, which is stored in secure storage on the user's device.

[0276] On the device: If necessary, the user or a relative can review the log file, which contains a portion of the call (where fraud is suspected) and details of the fraud determination.

[0277] Specific examples

[0278] One day, a user receives a call from someone claiming to be from a bank. As soon as the call begins, the device begins capturing the conversation in real time. A local AI model converts the voice data into text and analyzes the conversation. If the call contains keywords associated with fraud, such as "account number confirmation required," "unpaid bill," or "transfer," the emotion engine analyzes the user's tone of voice and word choice and recognizes that the user is confused. In this case, the device issues a warning message to the user: "Please remain calm. This call may be fraudulent. Do not continue the call." At the same time, a detailed email is sent to the user's relatives, stating, "The user is very confused. Please contact us immediately." After the user ends the call, the conversation and the fraud detection result are securely stored as a log. This process provides better protection for the elderly against telephone fraud.

[0279] The system performs real-time voice analysis as well as user emotion recognition, detecting fraud and taking appropriate action to provide peace of mind to seniors and their families.

[0280] The processing flow will be explained below.

[0281] Step 1:

[0282] User: The user installs the "Phone Fraud Prevention App" on their device. After installation, they launch the app and register on the initial setup screen. The user enters contact information for relatives and emergency notification methods to complete the setup.

[0283] Step 2:

[0284] Server: The server receives the user registration information and performs an authentication process. If authentication is successful, the user registration information is stored in a secure database.

[0285] Step 3:

[0286] User: When a call comes in to the user's phone, the user answers the call.

[0287] Step 4:

[0288] Terminal: When a call starts, the terminal starts capturing audio data in real time, which is temporarily stored in the terminal's buffer memory.

[0289] Step 5:

[0290] Device (local AI model): The speech recognition engine converts voice data into text data in real time. This conversion process is performed on the device, and the converted text data is sent to the analysis module.

[0291] Step 6:

[0292] On-device (local AI model): The analytics module receives the text data and filters it to determine whether it is potentially fraudulent, checking to see if specific fraud-related keywords or phrases are detected.

[0293] Step 7:

[0294] On-device (local AI model): Utilizing advanced natural language processing (NLP) to understand context and perform a detailed assessment of the likelihood of fraud. If fraud is deemed likely, a fraud alert flag is raised.

[0295] Step 8:

[0296] Device (Emotion Engine): Analyzes the user's emotional state using voice data. The emotion engine recognizes the user's emotional state (surprise, confusion, anger, etc.) based on the user's tone of voice and word choice.

[0297] Step 9:

[0298] Terminal: If the emotion engine determines that the user's emotion is negative, such as surprise or confusion, it generates a gentle warning message, such as "Please stay calm. This call may be fraudulent. Do not continue the call."

[0299] Step 10:

[0300] Terminal: If the fraud alert flag is activated, the terminal will audibly reproduce the generated warning message to the user.

[0301] Step 11:

[0302] Device: At the same time, promptly notify relatives, for example by sending an email or SMS with detailed information such as "User is receiving a scam call. This is particularly disturbing. Please contact them immediately," depending on the user's emotional state.

[0303] Step 12:

[0304] User: The user is warned and ends the call. Users who hear the warning are encouraged to end the call immediately.

[0305] Step 13:

[0306] Device: After the call is completed, the call details and fraud detection results are saved to a secure log file, which is stored in secure storage on the user's device.

[0307] Step 14:

[0308] Device: The user or a relative can check this log file as needed. The log file contains detailed records of suspected fraudulent calls and their evaluation results.

[0309] This provides real-time protection for seniors against phone scams, allows for flexible responses based on emotional state, and facilitates rapid response in emergencies by notifying relatives.

[0310] Example 2

[0311] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0312] To prevent elderly people from becoming victims of telephone fraud, a system that can quickly detect potential fraud and issue a warning is essential. However, conventional systems issue a uniform warning message without taking into account the user's emotional state, which can lead to users being unable to respond appropriately. Furthermore, notifications to relatives are also sent in a uniform manner, which creates an issue where the urgency of the situation is not accurately conveyed.

[0313] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring voice data in real time, means for converting the acquired voice data into text data, means for analyzing the possibility of fraud from the converted text data, means for issuing a warning when there is a possibility of fraud, means for recognizing the subscriber's emotion, means for adjusting the content of the warning depending on the recognized emotional state, and means for notifying relatives of the possibility of fraud. This enables flexible response according to the user's emotional state, effectively protecting the elderly from telephone fraud, and enabling accurate notification to relatives.

[0314] "Audio data" is data that represents an audio signal in digital form and is used to record the contents of a call.

[0315] "Real-time" refers to processing that occurs immediately without delay, meaning that voice data is acquired and processed immediately during a call.

[0316] "Text data" is data that expresses character information in digital format, and is obtained by converting voice data into a string of characters.

[0317] "Fraud likelihood" refers to assessing the likelihood that a call contains fraudulent activity and determining whether the risk is high or low.

[0318] A "warning" is a message provided by voice, text, or other means to alert the user when there is a high possibility of fraud.

[0319] "Subscribers" refer to individuals who use the system, which is primarily targeted at elderly people.

[0320] "Emotions" refer to an individual's psychological state or feelings, and specifically include states such as surprise, confusion, and anger.

[0321] "Emotional state" indicates the user's current emotional state, as analyzed by the emotion engine.

[0322] "Relatives" are family members or close relatives of the user, who are to be notified in the event of an emergency.

[0323] "Notification" is a means of conveying important information to others, and is sent to relatives in the form of email, SMS, etc.

[0324] MODE FOR CARRYING OUT THE INVENTION

[0325] System Overview

[0326] This invention is a system that performs real-time analysis of voice data and provides warnings and notifications to protect elderly people from becoming victims of telephone fraud. By combining this system with an emotion engine that recognizes the user's emotions, it enables flexible responses according to the user's emotional state.

[0327] Hardware and Software Configuration

[0328] 1. A means of acquiring audio data in real time

[0329] Device: The user's device is equipped with the capability to capture the contents of phone conversations in real time.

[0330] 2. A means of converting the acquired voice data into text data

[0331] Device (local AI model): The speech recognition engine converts voice data into text data in real time. This conversion process is performed within the device.

[0332] 3. A method for analyzing the possibility of fraud from the converted text data

[0333] On-device (local AI model): The analysis module analyzes the converted text data and filters it to detect fraudulent words and phrases. It uses natural language processing (NLP) techniques to understand the context and assess the likelihood of fraud.

[0334] 4. A way to flag potential fraud

[0335] Terminal: If a fraudulent call is detected, the terminal will issue an audio warning to the user, such as a warning message saying, "This call may be fraudulent. Do not continue the call."

[0336] 5. How to notify relatives of potential fraud

[0337] Server: The user's relatives are notified by email or SMS of a possible scam. This notification may include a message such as "The user has received a potentially scam call. Please check."

[0338] 6. Means of recognizing user emotions using an emotion engine

[0339] The device is equipped with an emotion engine that analyzes the user's emotional state based on voice data. The emotion engine analyzes the tone of voice and word choice to gauge the user's emotions (e.g., surprise, confusion, anger, etc.).

[0340] 7. A means for the emotion engine to tailor warning content based on the user's emotional state

[0341] Terminal: The emotion engine recognizes the user's emotional state and issues a warning message according to the user's emotion. For example, if the user is surprised, it will provide a gentle warning message such as "Please stay calm. This call may be a scam."

[0342] 8. A method for detailing notifications to relatives based on the user's emotional state

[0343] Device: If the user's emotional state exceeds a certain threshold, the notification to the relatives will be detailed. For example, if the user is very distressed, the relatives will receive a detailed message such as "The user is very distressed. Please contact us immediately."

[0344] Program processing explanation

[0345] Initial Setup

[0346] User: Install the "Phone Fraud Prevention App" on the device. After installation, register the contact information of relatives and the notification method in case of an emergency on the settings screen.

[0347] Server: Receives the user's registration information, performs the authentication process, and, if successful, stores the registration information in a secure database.

[0348] Real-time call analysis and emotion recognition

[0349] User: When a call comes in to the user's phone, answer the call.

[0350] Terminal: When a call starts, the terminal starts capturing voice data in real time, which is temporarily stored in the terminal's buffer memory.

[0351] Device (local AI model): The voice recognition engine converts voice data into text data in real time.

[0352] Terminal (local AI model): The analysis module receives the converted text data and filters it to analyze the possibility of fraud.

[0353] On-device (local AI model): The NLP engine understands the context and assesses the likelihood of fraud.

[0354] Terminal (emotion engine): Analyzes the user's emotional state based on voice data and recognizes emotions such as surprise, confusion, or anger.

[0355] Specific examples

[0356] Specific operation scenario

[0357] One day, the user receives a call from someone claiming to be from the bank. As soon as the call begins, the device begins capturing the conversation in real time. A local AI model converts the voice data into text and analyzes the conversation. If the call contains keywords associated with fraud, such as "account number confirmation required," "unpaid bill," or "transfer," the emotion engine analyzes the user's tone of voice and word choice and recognizes that the user is confused. In this case, the device issues a warning message to the user: "Please remain calm. This call may be fraudulent. Do not continue the call." At the same time, a detailed email is sent to the user's relatives stating, "The user is very confused. Please contact us immediately." After the user ends the call, the conversation and the fraud determination results are securely stored as logs.

[0358] Example prompts for generative AI models

[0359] "Generate natural language explanations for a system that utilizes an emotion engine to detect elderly phone scams in real time and issue appropriate warnings and notifications. The system analyzes voice data, recognizes emotions, alerts, and notifies."

[0360] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0361] Step 1: Initial Setup

[0362] User: The user installs the "Phone Fraud Prevention App" on their device. After installation is complete, they register the contact information of their relatives and the notification method in case of an emergency on the settings screen. The input data is the contact information of their relatives and the notification method, and the setting information is saved on the device as output.

[0363] Server: The server receives the registration information from the user and performs an authentication process. The input data is the user's registration information, and if the authentication is successful, the output is the registration information stored in a secure database.

[0364] Step 2: Detecting an incoming call

[0365] User: When a call comes in, the user answers the call.

[0366] Terminal: The terminal detects an incoming call and starts capturing the call content in real time when the call starts. The input data is the call start signal, and the output is the start of voice data capture. This operation is stored in buffer memory.

[0367] Step 3: Convert audio data to text

[0368] Terminal (local AI model): The speech recognition engine converts voice data during a call into text data in real time. The input data is voice data, and text data is generated as output. The specific operation is to apply a speech-to-text conversion algorithm.

[0369] Step 4: Fraud Potential Analysis

[0370] Terminal (Local AI Model): The analysis module receives the generated text data and filters out fraudulent words and phrases. The input data is text data, and the output is a determination of whether or not it is likely to be fraudulent. This filtering is performed using keyword search and NLP techniques.

[0371] Step 5: Emotion Recognition

[0372] Terminal (Emotion Engine): Analyzes the user's emotional state based on voice data. The input data is voice and text data, and the output recognizes the user's emotional state (surprise, confusion, anger, etc.). Specific operations include an emotion recognition algorithm that analyzes tone and word choice.

[0373] Step 6: Fraud detection

[0374] Terminal (local AI model): If it determines that there is a high probability of fraud, it activates a fraud alert flag. The input data are the filtering results and emotional state, and the output is a fraud alert flag.

[0375] Step 7: Sending an alert

[0376] Terminal: When the fraud alert flag is activated, a warning message is issued according to the user's emotional state as recognized by the emotion engine. The input data is the fraud alert flag and the emotional state, and the warning message is generated as the output. Specific operations include, for example, outputting a message such as "This call may be fraudulent. Please do not continue the call."

[0377] Step 8: Notify your next of kin

[0378] Terminal: If a fraud is deemed likely, a notification is sent to relatives via email or SMS. The input data is the fraud alert flag and the emotional state, and the output is a notification message. Specific actions include sending a detailed notification such as "The user is very confused. Please contact us immediately."

[0379] Step 9: End the call and save the log

[0380] User: Receives warning and ends call.

[0381] Terminal: After the call ends, the call content and fraud determination results are saved in a secure log file. The input data is the call content and fraud determination results, and log data is generated as output. This log data is stored in secure storage within the terminal. Specifically, the log file records part of the call content (the part suspected of being fraudulent) and details of the fraud determination.

[0382] In this way, the system can effectively protect the elderly from telephone fraud by performing real-time analysis combined with user emotion recognition.

[0383] (Application example 2)

[0384] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0385] This invention relates to a system for preventing elderly people from becoming victims of telephone fraud. In particular, it aims to enhance user safety by not only analyzing the possibility of fraud but also recognizing the user's emotions and responding flexibly according to their state. Current systems only detect fraud and do not take into account the user's emotional state, which often leaves elderly people confused and frightened. For this reason, there is a need for the development of a system that also incorporates emotion recognition.

[0386] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring voice data in real time, means for converting the acquired voice data into text data, means for analyzing the possibility of fraud from the converted text data, means for issuing a warning when there is a possibility of fraud, means for notifying relatives of the possibility of fraud, means for recognizing the user's emotions, means for adjusting the content of the warning depending on the user's emotional state, and means for detailing the content of the notification depending on the emotional state. This reduces the possibility that elderly people will become victims of telephone fraud and enables appropriate responses depending on the user's emotional state.

[0387] "Voice data" refers to data that is recorded or acquired in digital form, such as voice from a conversation.

[0388] "Real-time" means that the processing occurs immediately, without delay.

[0389] "Means" refers to a method or apparatus for achieving a particular function.

[0390] "Text data" is digital data that has been converted from voice data into character information.

[0391] "Fraud likelihood analysis" refers to assessing the likelihood of fraudulent activity based on voice and text data.

[0392] "Issuing a warning" refers to the act of issuing a warning or alert to the user.

[0393] "Relatives" refers to an individual's family or blood relatives, including parents, siblings, and children.

[0394] "Emotion recognition" means analyzing and determining a user's psychological state and emotions (for example, surprise, confusion, anger, etc.) from voice or text.

[0395] "Adjusting the warning content according to the emotional state" means generating a warning message with appropriate expressions and tone, taking into account the emotional state of the user.

[0396] "Detailing the notification content" means making the specific information and content of the notification to relatives more detailed and precise depending on the user's emotional state and the possibility of fraud.

[0397] "Local terminal" refers to a local computer or device that can be directly operated by a user.

[0398] A "local AI model" is an artificial intelligence model that is placed within a specific enclosure or device and operates independently of the internet.

[0399] This invention is a system to protect elderly people from becoming victims of telephone fraud. In particular, it not only analyzes the possibility of fraud but also recognizes the user's emotions and responds flexibly according to their emotional state.

[0400] System configuration

[0401] 1. Real-time acquisition of audio data

[0402] Terminal: Equipped with a function to capture voice data during a call in real time. Specifically, it uses the smartphone's microphone.

[0403] 2. Converting audio data to text data

[0404] On the device: Use the Google Speech-to-Text API to convert captured audio data into text data.

[0405] 3. Analyze for potential fraud

[0406] Terminal: Use spaCy as a natural language processing engine to analyze possible fraud from the converted text data.

[0407] 4. Emotion recognition

[0408] Terminal: Uses IBM Watson® Tone Analyzer to analyze the user's emotional state based on voice and text data.

[0409] 5. Alerts and Notifications

[0410] Terminal: If there is a high possibility of fraud, the system will warn the user and send a notification to their relatives. The warning message will be generated using the results of the emotion engine.

[0411] Server: To notify relatives, we use Twilio to send SMS and SMTP to send email.

[0412] 6. Log storage

[0413] On the device: Call logs and analytics are stored in a SQLite database for secure use.

[0414] Specific examples

[0415] One day, the user receives a call on their smartphone from someone claiming to be from the bank. As soon as the call begins, the device begins capturing the call in real time, and the audio data is converted into text using the Google Speech-to-Text API. If the converted text contains keywords that suggest fraud, such as "account number confirmation required," "unpaid balance," or "transfer," the device analyzes the text data with spaCy and determines that there is a high possibility of fraud. At the same time, the device uses IBM Watson Tone Analyzer to analyze the user's emotional state and recognizes that the user is feeling confused or surprised.

[0416] Based on the results, the device will display a soft warning message to the user, such as "Please stay calm. This call may be fraudulent. Do not continue the call." Additionally, Twilio will be used to send a detailed email notification to the next of kin, such as "The user is very distressed. Please contact us immediately."

[0417] This series of processes is extremely effective in strengthening user security and preventing fraud before it occurs. In addition, the information saved as a log can be used for future verification and as evidence.

[0418] Example prompts for generative AI models

[0419] Write a program that converts voice data into text data, analyzes the text data, and detects possible fraud. This program uses IBM Watson Tone Analyzer to recognize the user's emotions, and if fraud is detected, sends an SMS to relatives using the Twilio API and an email using SMTP. Also, save the call content and analysis results in a SQLite database.

[0420] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0421] Step 1:

[0422] The device acquires voice data in real time. This involves using the smartphone's microphone to capture voice data from the beginning to the end of a call. The input is the voice during the call, and the output is the captured voice data.

[0423] Step 2:

[0424] The device converts the acquired voice data into text data. The Google Speech-to-Text API is used to convert the voice data into text format. The input is voice data and the output is text data. This conversion allows the spoken content to be handled as text information.

[0425] Step 3:

[0426] The device analyzes the converted text data to detect potential fraud. It uses a natural language processing engine (e.g., spaCy) to assess the likelihood of fraud based on specific keywords and phrases. The input is the text data, and the output is a fraud assessment result (whether fraud is likely or unlikely). This analysis identifies indicators of fraudulent activity.

[0427] Step 4:

[0428] An emotion recognition engine is used to analyze the user's emotional state. IBM Watson Tone Analyzer is used to determine the user's emotion (for example, surprise, confusion, anger, etc.) based on text data and voice tone. The input is voice data and text data, and the output is the result of the emotional state determination. This allows the user's psychological state to be understood.

[0429] Step 5:

[0430] If the device is determined to be highly likely to be a fraudster, a warning is issued to the user. Based on the emotion recognition results, a warning message is generated in an appropriate tone. For example, it could say, "This call may be fraudulent. Please do not continue the call." The input is the fraud detection result and the emotional state detection result, and the output is the warning message. This allows the user to be immediately alerted.

[0431] Step 6:

[0432] The device notifies relatives of possible fraud. It uses the Twilio API to send an SMS and an email via SMTP. This notification contains detailed information based on the user's emotional state. For example, it could say, "The user is very confused. Please contact us immediately." The input is the fraud detection result and the emotional state detection result, and the output is the SMS and email sent. This allows relatives to respond quickly.

[0433] Step 7:

[0434] The device saves the call content and analysis results in a secure log file. An SQLite database is used to record the call content, fraud detection results, and emotional state detection results. The input is the text data of the call content, fraud detection results, and emotional state detection results, and the output is the saved log file. This makes it possible to check the call content and fraud detection results later.

[0435] Step 8:

[0436] The user or a relative can check the log file as needed. The log file contains a portion of the call content (parts suspected of fraud) and the analysis results. The input is the request, and the output is the display of the log file. This allows the user to check the details of past call content and the fraud detection status.

[0437] Prompt Sentence Examples

[0438] Write a program that converts voice data into text data, analyzes the text data, and detects possible fraud. This program uses IBM Watson Tone Analyzer to recognize the user's emotions, and if fraud is detected, sends an SMS to relatives using the Twilio API and an email using SMTP. Also, save the call content and analysis results in a SQLite database.

[0439] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0440] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0441] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0442] [Second embodiment]

[0443] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0444] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0445] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0446] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0447] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0448] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0449] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0450] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0451] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0452] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0453] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0454] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0455] This invention is a system for analyzing voice data in real time and providing warnings and notifications, designed to prevent elderly people from becoming victims of telephone fraud. The system of the present invention has the following configuration in particular.

[0456] System configuration

[0457] 1. A means of acquiring audio data in real time

[0458] Device: The user's device is equipped with real-time audio capture capabilities, which allows for instantaneous capture of phone conversations.

[0459] 2. A means of converting the acquired voice data into text data

[0460] Terminal (local AI model): Equipped with a speech recognition engine, it converts call content into text data in real time, allowing voice data to be handled as text information.

[0461] 3. A method for analyzing the possibility of fraud from the converted text data

[0462] Device (Local AI Model): Natural Language Processing (NLP) techniques are used to analyze the converted text data and detect fraudulent words and phrases. The local AI model determines the likelihood of fraud based on known fraud techniques.

[0463] 4. A way to flag potential fraud

[0464] Device: If a potential scam is detected, the device will immediately issue an audio warning to the user, including specific instructions such as "This call may be fraudulent. Do not continue the call."

[0465] 5. How to notify relatives of potential fraud

[0466] Server: If a potential scam is detected, the user's relatives will be notified via email or SMS, with a message such as "The user has received a potentially fraudulent call. Please check."

[0467] Program processing

[0468] Initial Setup

[0469] User: The user installs the "Phone Fraud Prevention App" (hereinafter referred to as the "App") on their device. After installation, they register the contact information of their relatives on a simple setup screen.

[0470] Server: The server receives user registration information and performs authentication and configuration.

[0471] Real-time call analysis

[0472] User: When a user answers the phone, the call begins.

[0473] Terminal: When a call begins, the terminal begins capturing voice data in real time.

[0474] Device (local AI model): The speech recognition engine converts voice data into text in real time.

[0475] On-device (local AI model): Analyzes the converted text data and filters it to see if it contains malicious words or phrases.

[0476] On-device (local AI model): Uses advanced natural language processing capabilities to understand context and assess the likelihood of fraud.

[0477] Fraud detection and notification

[0478] Device (local AI model): Activate fraud alert flag if fraud is determined to be highly likely.

[0479] Device: If the fraud alert flag is activated, the user will hear a voice warning saying, "This call may be fraudulent. Do not continue."

[0480] Device: At the same time, your relatives will be notified by email or SMS of the possible fraud.

[0481] Ending a call and saving a log

[0482] User: The user ends the call.

[0483] Device: After the call is completed, the call details and fraud detection results are saved in a secure log file.

[0484] Device: Saved logs are stored in secure storage on the user's device and can be viewed by the user or their relatives if necessary.

[0485] Specific examples

[0486] One day, the user receives a call on their mobile phone from someone claiming to be from a "public institution." As soon as the call begins, the device begins capturing the call content in real time. The local AI model converts the voice data into text and analyzes the call content. If the call content contains keywords associated with fraud, such as "unpaid bills" or "transfers," the local AI model determines that it is likely a scam. The device then immediately issues a voice warning to the user and simultaneously sends a notification to their relatives. When the call ends, the suspected fraudulent call content and the fraud determination result are recorded in a log and securely stored. This series of processes significantly reduces the risk of elderly people falling victim to fraud.

[0487] The system can provide peace of mind to seniors and their families through real-time voice analysis and immediate response.

[0488] The processing flow will be explained below.

[0489] Step 1:

[0490] User: The user installs the "Phone Fraud Prevention App" on their device. After installation, the initial setup screen is displayed. The user enters the contact information of their relatives and completes the setup.

[0491] Step 2:

[0492] Server: The server receives the user registration information and performs an authentication process. If authentication is successful, the server stores the registration information in a secure database.

[0493] Step 3:

[0494] User: When a call comes in to the user's phone, the user answers the call.

[0495] Step 4:

[0496] Terminal: When a call starts, the terminal starts capturing audio data in real time, which is temporarily stored in the terminal's buffer memory.

[0497] Step 5:

[0498] Device (local AI model): The speech recognition engine converts voice data into text data in real time. This conversion process is performed on the device, and the converted text data is sent to the analysis module.

[0499] Step 6:

[0500] On-device (local AI model): The analysis module receives the converted text data and filters it to analyze potential fraud. If specific keywords or phrases are detected, the content is analyzed in detail.

[0501] Step 7:

[0502] On-device (local AI model): An advanced natural language processing (NLP) engine understands the context and assesses the likelihood of fraud from text data. If fraud is deemed likely, a fraud alert flag is raised.

[0503] Step 8:

[0504] Terminal: When the fraud alert flag is activated, the terminal will immediately issue an audio warning to the user, playing a specific warning message such as "This call may be fraudulent. Do not continue the call."

[0505] Step 9:

[0506] At the same time, the device will send emails and / or SMS to registered contacts of relatives informing them of the potential fraud, including a brief description of the alleged fraud.

[0507] Step 10:

[0508] User: The user receives a warning and ends the call. It is recommended that the call be ended immediately after the warning.

[0509] Step 11:

[0510] Device: After the call ends, the device saves the call details and fraud detection results in a secure log file that is stored in secure storage on the user's device.

[0511] Step 12:

[0512] Device: The user or a relative can review this log file if necessary. The log file contains a portion of the call (suspected fraudulent portion) and details of the fraud determination.

[0513] This series of processing steps results in a system that provides real-time protection for seniors against telephone fraud.

[0514] Example 1

[0515] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0516] As the elderly are increasingly vulnerable to telephone fraud, there is a need to provide a safe calling environment. Conventional countermeasures against telephone fraud are mostly reactive and do not provide real-time fraud prevention measures. This means that the risk of elderly people falling victim to fraud and suffering financial and psychological damage cannot be fully prevented.

[0517] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0518] In this invention, the server includes means for acquiring voice data in real time, means for converting the acquired voice data into text data, means for analyzing the possibility of fraud from the converted text data, means for issuing a warning when there is a possibility of fraud, means for notifying relatives of the possibility of fraud, means for saving data after the call ends, and means for storing the analyzed data in secure storage. This reduces the risk of elderly people falling victim to telephone fraud in real time and enables immediate contact with relatives, thereby providing a safe calling environment.

[0519] "Voice data" means information in analog or digital form that contains the human voice.

[0520] "Real-time" means that processing occurs immediately, without delay.

[0521] A "capturing means" is a method or device for capturing or collecting audio data.

[0522] "Text data" is voice data expressed as text information.

[0523] A "converting means" is a method or device for converting audio data into text data.

[0524] The "analyzing means" refers to a method or device for analyzing the converted text data to detect specific patterns or keywords.

[0525] "Probability of fraud" is the probability that a communication is deemed to contain fraud or fraudulent intent.

[0526] "Warning means" refers to a method or device for sending a sound or message to alert the user.

[0527] "Means of notification" refers to the method or device used to communicate information to relatives.

[0528] "Means for saving data after a call ends" refers to a method or device for saving the contents of the call and the analysis results.

[0529] "Secure storage" refers to storage environments and technologies that ensure data safety.

[0530] "Local terminal" refers to a device or equipment that is directly operated by a user.

[0531] A "local AI model" is an artificial intelligence technology that runs within a device and performs specific analyses and judgments.

[0532] A "means for detecting the start of a call" is a method or device for recognizing that a call has been received or started.

[0533] This invention is a system for preventing elderly people from becoming victims of telephone fraud, and performs real-time analysis of voice data, warnings, and notifications. The detailed configuration and operation of this system will be described below.

[0534] System configuration

[0535] 1. A means of acquiring audio data in real time

[0536] Device: The user's device is equipped with a built-in microphone to capture audio data in real time, in order to detect when a call starts and instantly capture the contents of the call.

[0537] 2. A means of converting the acquired voice data into text data

[0538] Terminal (local AI model): Equipped with a speech recognition engine (e.g., Google Speech-to-Text or Microsoft Azure Cognitive Services), it converts the contents of the call into text data in real time, allowing the voice data to be treated as text information.

[0539] 3. A method for analyzing the possibility of fraud from the converted text data

[0540] Terminal (local AI model): The analysis engine uses natural language processing (NLP) technology to analyze the converted text data, detecting fraudulent words and phrases based on existing fraud techniques and determining the likelihood of fraud.

[0541] 4. A way to flag potential fraud

[0542] Device: If a call is deemed potentially fraudulent, the device will immediately issue an audio warning to the user, including messages such as "This call may be fraudulent. Do not continue the call."

[0543] 5. How to notify relatives of potential fraud

[0544] Server: If a potential scam is detected, the user's relatives will be notified via email and / or SMS, with a message such as "The user has received a potentially fraudulent call. Please check."

[0545] 6. How to save data after a call ends

[0546] On the device: After the call ends, the analyzed text data and the log file containing the fraud detection result are stored in a secure storage location. This storage process is performed in a secure manner so that it can be accessed by the user.

[0547] 7. A means of storing the analyzed data in secure storage

[0548] Device: Saved log files are stored in secure storage on the device and can be viewed by the user or their relatives if necessary.

[0549] Specific examples

[0550] One day, the user receives a call on their mobile phone from someone claiming to be from a public institution. As soon as the call begins, the device begins capturing the call content in real time. The local AI model converts the voice data into text and analyzes the text data. If the call content contains keywords associated with fraud, such as "unpaid bills" or "transfers," the local AI model determines that it is likely a scam. The device then immediately issues an audio warning to the user and simultaneously sends a notification to their relatives. After the call ends, the suspected fraudulent content of the call and the fraud determination result are recorded in a log file and securely stored. This series of processes significantly reduces the risk of elderly people falling victim to fraud.

[0551] Prompt Sentence Examples

[0552] "Please explain the system to be followed when an elderly person receives a fraudulent phone call."

[0553] The system can provide peace of mind to seniors and their families through real-time voice analysis and immediate response.

[0554] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0555] Step 1:

[0556] When a user answers a call, the call begins. The device detects the start of the call and begins capturing audio data using the built-in microphone. The input at this time is the audio signal during the call, and the output is audio data. Specifically, the built-in microphone of the device collects the telephone voice and converts it into digital audio data.

[0557] Step 2:

[0558] The device converts the captured voice data into text data in real time. This uses a local AI model speech recognition engine (e.g., Google Speech-to-Text or Microsoft Azure Cognitive Services). The input is voice data, and the output is text data. Specifically, the speech recognition engine analyzes voice patterns and converts them into corresponding text data.

[0559] Step 3:

[0560] The device analyzes the text data to assess the likelihood of fraud. A local AI model, the Natural Language Processing (NLP) engine, is responsible for this process. The input is the text data, and the output is a flag indicating the likelihood of fraud (a fraud alert flag). Specifically, the NLP engine filters keywords and phrases in the text to detect patterns associated with fraud.

[0561] Step 4:

[0562] When the fraud alert flag is activated, the terminal immediately issues an audio warning to the user. The input is the fraud alert flag and the output is the warning message. Specifically, the terminal plays the audio message "This call may be fraudulent. Do not continue the call."

[0563] Step 5:

[0564] When the fraud alert flag is activated, the server notifies the relative of a possible fraud. The input is a signal sent from the device to the server, and the output is a notification message to the relative. Specifically, the server sends an email or SMS to the relative's registered contacts saying, "A user has answered a suspected fraudulent call. Please check."

[0565] Step 6:

[0566] When the user ends the call, the device saves the call content and the fraud detection result in a secure log file. The input is a call end signal and text data containing the fraud detection result, and the output is a secure log file. Specifically, the device encrypts the data and saves it in secure storage.

[0567] Step 7:

[0568] The saved log file is stored in secure storage within the device. The input is a secure log file, and the output is saved in secure storage. Specifically, the device saves the log file according to a specified security protocol and allows the user or relatives to access it as needed.

[0569] (Application example 1)

[0570] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0571] There is a need to effectively reduce the risk of elderly people becoming victims of telephone fraud. However, it is difficult for elderly people to recognize the signs of fraud themselves, and they are unable to act quickly without appropriate warnings and notifications. To solve these problems, a system that processes voice data in real time and provides immediate warnings is required.

[0572] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0573] In this invention, the server includes a means for acquiring voice data in real time, a means for converting the acquired voice data into text data, a means for analyzing the possibility of fraud from the converted text data, a means for notifying relatives of the possibility of fraud if there is a possibility of fraud, and a means for issuing visual and audio warnings, thereby significantly reducing the risk of elderly people becoming victims of telephone fraud and enabling prompt action.

[0574] "Audio data" refers to recordings of voice during a call, etc., and is information that is subject to signal processing.

[0575] "Means of acquisition in real time" refers to technical means for collecting voice data immediately without delay.

[0576] "Text data" is voice data converted into character information, and is basic data for analysis.

[0577] "Means for converting" refers to the speech recognition technology used to convert voice data into text data.

[0578] "Means for analyzing" refers to algorithms and techniques used to process the captured and converted text data to assess the likelihood of fraud.

[0579] "Alert Measures" refers to the audio and visual interfaces used to notify the user of potential fraud.

[0580] "Means of notifying relatives of possible fraud" refers to notification techniques that use email, SMS, or other means to inform a user's relatives of possible fraud.

[0581] "Visual and audio warning means" refers to devices and technologies that display and transmit visual and audio warnings that are intuitively understandable to the user.

[0582] A "local terminal" is a device distributed to a user, and is an apparatus that has computational resources and storage for real-time processing.

[0583] "Local AI model" refers to an artificial intelligence model that runs within the device and incorporates learning algorithms to detect signs of fraud.

[0584] "Means for detecting the start of a call" refers to a technical means for recognizing that a telephone call has started and immediately starting to capture voice data.

[0585] This invention is a system for analyzing voice data in real time and providing warnings and notifications, designed to prevent elderly people from becoming victims of telephone fraud. The system of the present invention has the following configuration in particular.

[0586] System configuration

[0587] The system includes means for acquiring voice data in real time, means for converting the acquired voice data into text data, means for analyzing the converted text data for the possibility of fraud, means for issuing an alert when there is a possibility of fraud, means for notifying relatives of the possibility of fraud, and means for issuing visual and audio alerts, thereby enabling the user to immediately recognize the risk of fraud and the relatives to quickly respond to the crisis.

[0588] Program processing

[0589] Initial Setup

[0590] User: The user installs the "Phone Fraud Prevention App" on their device. After installation, they register the contact information of their relatives on a simple setup screen.

[0591] Server: The server receives user registration information and performs authentication and configuration.

[0592] Real-time call analysis

[0593] Terminal (smart glasses): When a call starts, the microphone built into the smart glasses captures voice data in real time.

[0594] Terminal: The acquired voice data is converted into text data using a voice recognition engine (e.g., Google API).

[0595] On-device (local AI model): Analyzes the converted text data to see if it contains fraudulent words or phrases. This uses the ScamPredictionModel.

[0596] Fraud detection and notification

[0597] Device (local AI model): Activate fraud alert flag if fraud is determined to be highly likely.

[0598] Device: If the fraud alert flag is activated, the user will be warned audibly, "This call may be fraudulent. Do not continue the call." A visual warning will also be displayed on the smart glasses display.

[0599] Server: At the same time, notify the relatives of the possible fraud via email or SMS (using the smtplib library).

[0600] Ending a call and saving a log

[0601] User: The user ends the call.

[0602] Device: After the call is completed, the call details and fraud detection results are saved in a secure log file.

[0603] On the device: The saved logs are stored in secure storage and can be viewed by the user or a relative if necessary.

[0604] Specific examples

[0605] One day, the user receives a call through their smart glasses from someone claiming to be from a government agency. As soon as the call begins, the smart glasses begin capturing the call content in real time. The local AI model converts the audio data into text and analyzes the call content. If the call content contains keywords associated with fraud, such as "unpaid bills" or "transfers," the local AI model determines that it is likely a scam. The smart glasses then immediately issue an audio and visual warning to the user and simultaneously send an emergency notification to their relatives. When the call ends, the suspected fraudulent call content and the fraud determination result are logged and securely stored.

[0606] Generative AI model prompt example

[0607] "Develop an AI model to identify potentially fraudulent phone call content. The model will take Japanese text data as input and determine whether it is likely to be fraudulent. Examples of phrases include: "unpaid money", "transfer". The output should be a Boolean value indicating the likelihood of fraud."

[0608] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0609] Step 1:

[0610] User initiates a call

[0611] Input: A user receives a call through their smart glasses.

[0612] How it works: The smart glasses' call start detection sensor recognizes the start of a call and starts collecting voice data.

[0613] Output: A call initiation signal is sent to the processing unit of the smart glasses.

[0614] Step 2:

[0615] Acquiring audio data

[0616] Input: Call initiation signal and call audio.

[0617] How it works: The smart glasses have a built-in microphone that captures audio data in real time, which is then converted into digital form and sent to a processing unit.

[0618] Output: Digital signal of audio data.

[0619] Step 3:

[0620] Converting audio data to text

[0621] Input: Digital signal of audio data.

[0622] How it works: Using a speech recognition engine (e.g. Google API) within the smart glasses, the voice data is converted into text data. During this conversion process, the voice signal is analyzed and converted into corresponding text information.

[0623] Output: Text data.

[0624] Step 4:

[0625] Text data analysis

[0626] Input: Text data.

[0627] How it works: A local AI model (ScamPredictionModel) is used to analyze text data. The analysis process detects keywords and context within the text data and assesses its likelihood of fraud.

[0628] Output: A flag indicating whether fraud is likely.

[0629] Step 5:

[0630] Fraud Alert Verification

[0631] Input: Possible fraud flag.

[0632] How it works: If the fraud alert flag is active, the system determines that fraud is likely.

[0633] Output: Alert signal.

[0634] Step 6:

[0635] Sending a warning to the user

[0636] Input: Alert signal.

[0637] How it works: The smart glasses provide users with visual and audio warnings, including a message on the display that reads "This call may be fraudulent. Do not continue the call," as well as an audio warning.

[0638] Output: A warning notice to the user.

[0639] Step 7:

[0640] Sending notifications to relatives

[0641] Input: Alert signal.

[0642] What it does: The server sends emergency notifications via email or SMS based on the contact information of the next of kin configured. It uses the smtplib library to send the notifications.

[0643] Output: Notification sent to relatives.

[0644] Step 8:

[0645] Call content and result logging

[0646] Input: call termination signal, call content, fraud detection result.

[0647] How it works: When a user ends a call, the device saves the call and the fraud detection result in a secure log file that is encrypted and stored in secure storage on the user's device.

[0648] Output: Secure log file.

[0649] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0650] This invention is a system that performs real-time analysis of voice data and provides warnings and notifications to protect elderly people from becoming victims of telephone fraud. In particular, by combining this system with an emotion engine that recognizes the user's emotions, this system enables flexible responses according to the user's emotional state.

[0651] System configuration

[0652] 1. A means of acquiring audio data in real time

[0653] Device: The user's device is equipped with the capability to capture phone conversations in real time.

[0654] 2. A means of converting the acquired voice data into text data

[0655] Device (local AI model): A speech recognition engine converts voice data into text data in real time. This conversion process is performed within the device.

[0656] 3. A method for analyzing the possibility of fraud from the converted text data

[0657] On-device (local AI model): The analytics module analyzes the converted text data and filters it to detect fraudulent words and phrases. It uses natural language processing (NLP) techniques to understand the context and assess the likelihood of fraud.

[0658] 4. A way to flag potential fraud

[0659] Terminal: If a call is deemed potentially fraudulent, the terminal will issue an audio warning to the user, such as a warning message saying, "This call may be fraudulent. Do not continue the call."

[0660] 5. How to notify relatives of potential fraud

[0661] Server: Notify the user's relatives of the potential scam via email or SMS. This notification may include a message such as "The user has received a potentially fraudulent call. Please check."

[0662] 6. Means of recognizing user emotions using an emotion engine

[0663] The device is equipped with an emotion engine that analyzes the user's emotional state based on voice data. The emotion engine analyzes the tone of voice and word choice to gauge the user's emotions (e.g., surprise, confusion, anger, etc.).

[0664] 7. A means for the emotion engine to tailor warning content based on the user's emotional state

[0665] Terminal: The emotion engine recognizes the user's emotional state and issues a warning message according to the user's emotion. For example, if the user is surprised, it will provide a gentle warning message such as "Please stay calm. This call may be a scam."

[0666] 8. A method for detailing notifications to relatives based on the user's emotional state

[0667] Device: If the user's emotional state exceeds a certain threshold, the notification to the next of kin will be detailed. For example, if the user is very distressed, the next of kin will receive a detailed message saying, "The user is very distressed. Please contact us immediately."

[0668] Program processing

[0669] Initial Setup

[0670] User: The user installs the "Phone Fraud Prevention App" on their device. After installation, they register the contact information of their relatives and the method of notification in case of an emergency on the settings screen.

[0671] Server: The server receives the user registration information, performs an authentication process, and, if successful, stores the registration information in a secure database.

[0672] Real-time call analysis and emotion recognition

[0673] User: When a call comes in to the user's phone, answer the call.

[0674] Terminal: When a call starts, the terminal starts capturing audio data in real time, which is temporarily stored in the terminal's buffer memory.

[0675] Device (local AI model): The speech recognition engine converts voice data into text data in real time.

[0676] Terminal (local AI model): The analysis module receives the converted text data and filters it to analyze possible fraud.

[0677] On-device (local AI model): A natural language processing engine understands the context and assesses the likelihood of fraud.

[0678] Terminal (emotion engine): Analyzes the user's emotional state based on voice data and recognizes emotions such as surprise, confusion, or anger.

[0679] Fraud detection and emotion-based warnings and notifications

[0680] Device (local AI model): Activate fraud alert flag if fraud is determined to be highly likely.

[0681] Terminal: When the fraud alert flag is activated, the emotion engine recognizes the user's emotional state and issues a warning message, such as "This call may be fraudulent. Do not continue the call."

[0682] On the device: At the same time, an email or SMS is sent to a relative notifying them of the possible scam. If the user's emotional state is very distressed, the notification will include a detailed message such as "The user is very distressed. Please contact us immediately."

[0683] Ending a call and saving a log

[0684] User: The user receives an alert and ends the call.

[0685] Device: After the call is completed, the call details and fraud detection results are saved in a secure log file, which is stored in secure storage on the user's device.

[0686] On the device: If necessary, the user or a relative can review the log file, which contains a portion of the call (where fraud is suspected) and details of the fraud determination.

[0687] Specific examples

[0688] One day, a user receives a call from someone claiming to be from a bank. As soon as the call begins, the device begins capturing the conversation in real time. A local AI model converts the voice data into text and analyzes the conversation. If the call contains keywords associated with fraud, such as "account number confirmation required," "unpaid bill," or "transfer," the emotion engine analyzes the user's tone of voice and word choice and recognizes that the user is confused. In this case, the device issues a warning message to the user: "Please remain calm. This call may be fraudulent. Do not continue the call." At the same time, a detailed email is sent to the user's relatives, stating, "The user is very confused. Please contact us immediately." After the user ends the call, the conversation and the fraud detection result are securely stored as a log. This process provides better protection for the elderly against telephone fraud.

[0689] The system performs real-time voice analysis as well as user emotion recognition, detecting fraud and taking appropriate action to provide peace of mind to seniors and their families.

[0690] The processing flow will be explained below.

[0691] Step 1:

[0692] User: The user installs the "Phone Fraud Prevention App" on their device. After installation, they launch the app and register on the initial setup screen. The user enters contact information for relatives and emergency notification methods to complete the setup.

[0693] Step 2:

[0694] Server: The server receives the user registration information and performs an authentication process. If authentication is successful, the user registration information is stored in a secure database.

[0695] Step 3:

[0696] User: When a call comes in to the user's phone, the user answers the call.

[0697] Step 4:

[0698] Terminal: When a call starts, the terminal starts capturing audio data in real time, which is temporarily stored in the terminal's buffer memory.

[0699] Step 5:

[0700] Device (local AI model): The speech recognition engine converts voice data into text data in real time. This conversion process is performed on the device, and the converted text data is sent to the analysis module.

[0701] Step 6:

[0702] On-device (local AI model): The analytics module receives the text data and filters it to determine whether it is potentially fraudulent, checking to see if specific fraud-related keywords or phrases are detected.

[0703] Step 7:

[0704] On-device (local AI model): Utilizing advanced natural language processing (NLP) to understand context and perform a detailed assessment of the likelihood of fraud. If fraud is deemed likely, a fraud alert flag is raised.

[0705] Step 8:

[0706] Device (Emotion Engine): Analyzes the user's emotional state using voice data. The emotion engine recognizes the user's emotional state (surprise, confusion, anger, etc.) based on the user's tone of voice and word choice.

[0707] Step 9:

[0708] Terminal: If the emotion engine determines that the user's emotion is negative, such as surprise or confusion, it generates a gentle warning message, such as "Please stay calm. This call may be fraudulent. Do not continue the call."

[0709] Step 10:

[0710] Terminal: If the fraud alert flag is activated, the terminal will audibly reproduce the generated warning message to the user.

[0711] Step 11:

[0712] Device: At the same time, promptly notify relatives, for example by sending an email or SMS with detailed information such as "User is receiving a scam call. This is particularly disturbing. Please contact them immediately," depending on the user's emotional state.

[0713] Step 12:

[0714] User: The user is warned and ends the call. Users who hear the warning are encouraged to end the call immediately.

[0715] Step 13:

[0716] Device: After the call is completed, the call details and fraud detection results are saved to a secure log file, which is stored in secure storage on the user's device.

[0717] Step 14:

[0718] Device: The user or a relative can check this log file as needed. The log file contains detailed records of suspected fraudulent calls and their evaluation results.

[0719] This provides real-time protection for seniors against phone scams, allows for flexible responses based on emotional state, and facilitates rapid response in emergencies by notifying relatives.

[0720] Example 2

[0721] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0722] To prevent elderly people from becoming victims of telephone fraud, a system that can quickly detect potential fraud and issue a warning is essential. However, conventional systems issue a uniform warning message without taking into account the user's emotional state, which can lead to users being unable to respond appropriately. Furthermore, notifications to relatives are also sent in a uniform manner, which creates an issue where the urgency of the situation is not accurately conveyed.

[0723] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring voice data in real time, means for converting the acquired voice data into text data, means for analyzing the possibility of fraud from the converted text data, means for issuing a warning when there is a possibility of fraud, means for recognizing the subscriber's emotion, means for adjusting the content of the warning depending on the recognized emotional state, and means for notifying relatives of the possibility of fraud. This enables flexible response according to the user's emotional state, effectively protecting the elderly from telephone fraud, and enabling accurate notification to relatives.

[0724] "Audio data" is data that represents an audio signal in digital form and is used to record the contents of a call.

[0725] "Real-time" refers to processing that occurs immediately without delay, meaning that voice data is acquired and processed immediately during a call.

[0726] "Text data" is data that expresses character information in digital format, and is obtained by converting voice data into a string of characters.

[0727] "Fraud likelihood" refers to assessing the likelihood that a call contains fraudulent activity and determining whether the risk is high or low.

[0728] A "warning" is a message provided by voice, text, or other means to alert the user when there is a high possibility of fraud.

[0729] "Subscribers" refer to individuals who use the system, which is primarily targeted at elderly people.

[0730] "Emotions" refer to an individual's psychological state or feelings, and specifically include states such as surprise, confusion, and anger.

[0731] "Emotional state" indicates the user's current emotional state, as analyzed by the emotion engine.

[0732] "Relatives" are family members or close relatives of the user, who are to be notified in the event of an emergency.

[0733] "Notification" is a means of conveying important information to others, and is sent to relatives in the form of email, SMS, etc.

[0734] MODE FOR CARRYING OUT THE INVENTION

[0735] System Overview

[0736] This invention is a system that performs real-time analysis of voice data and provides warnings and notifications to protect elderly people from becoming victims of telephone fraud. By combining this system with an emotion engine that recognizes the user's emotions, it enables flexible responses according to the user's emotional state.

[0737] Hardware and Software Configuration

[0738] 1. A means of acquiring audio data in real time

[0739] Device: The user's device is equipped with the capability to capture the contents of phone conversations in real time.

[0740] 2. A means of converting the acquired voice data into text data

[0741] Device (local AI model): The speech recognition engine converts voice data into text data in real time. This conversion process is performed within the device.

[0742] 3. A method for analyzing the possibility of fraud from the converted text data

[0743] On-device (local AI model): The analysis module analyzes the converted text data and filters it to detect fraudulent words and phrases. It uses natural language processing (NLP) techniques to understand the context and assess the likelihood of fraud.

[0744] 4. A way to flag potential fraud

[0745] Terminal: If a fraudulent call is detected, the terminal will issue an audio warning to the user, such as a warning message saying, "This call may be fraudulent. Do not continue the call."

[0746] 5. How to notify relatives of potential fraud

[0747] Server: The user's relatives are notified by email or SMS of a possible scam. This notification may include a message such as "The user has received a potentially scam call. Please check."

[0748] 6. Means of recognizing user emotions using an emotion engine

[0749] The device is equipped with an emotion engine that analyzes the user's emotional state based on voice data. The emotion engine analyzes the tone of voice and word choice to gauge the user's emotions (e.g., surprise, confusion, anger, etc.).

[0750] 7. A means for the emotion engine to tailor warning content based on the user's emotional state

[0751] Terminal: The emotion engine recognizes the user's emotional state and issues a warning message according to the user's emotion. For example, if the user is surprised, it will provide a gentle warning message such as "Please stay calm. This call may be a scam."

[0752] 8. A method for detailing notifications to relatives based on the user's emotional state

[0753] Device: If the user's emotional state exceeds a certain threshold, the notification to the relatives will be detailed. For example, if the user is very distressed, the relatives will receive a detailed message such as "The user is very distressed. Please contact us immediately."

[0754] Program processing explanation

[0755] Initial Setup

[0756] User: Install the "Phone Fraud Prevention App" on the device. After installation, register the contact information of relatives and the notification method in case of an emergency on the settings screen.

[0757] Server: Receives the user's registration information, performs the authentication process, and, if successful, stores the registration information in a secure database.

[0758] Real-time call analysis and emotion recognition

[0759] User: When a call comes in to the user's phone, answer the call.

[0760] Terminal: When a call starts, the terminal starts capturing voice data in real time, which is temporarily stored in the terminal's buffer memory.

[0761] Device (local AI model): The voice recognition engine converts voice data into text data in real time.

[0762] Terminal (local AI model): The analysis module receives the converted text data and filters it to analyze the possibility of fraud.

[0763] On-device (local AI model): The NLP engine understands the context and assesses the likelihood of fraud.

[0764] Terminal (emotion engine): Analyzes the user's emotional state based on voice data and recognizes emotions such as surprise, confusion, or anger.

[0765] Specific examples

[0766] Specific operation scenario

[0767] One day, the user receives a call from someone claiming to be from the bank. As soon as the call begins, the device begins capturing the conversation in real time. A local AI model converts the voice data into text and analyzes the conversation. If the call contains keywords associated with fraud, such as "account number confirmation required," "unpaid bill," or "transfer," the emotion engine analyzes the user's tone of voice and word choice and recognizes that the user is confused. In this case, the device issues a warning message to the user: "Please remain calm. This call may be fraudulent. Do not continue the call." At the same time, a detailed email is sent to the user's relatives stating, "The user is very confused. Please contact us immediately." After the user ends the call, the conversation and the fraud determination results are securely stored as logs.

[0768] Example prompts for generative AI models

[0769] "Generate natural language explanations for a system that utilizes an emotion engine to detect elderly phone scams in real time and issue appropriate warnings and notifications. The system analyzes voice data, recognizes emotions, alerts, and notifies."

[0770] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0771] Step 1: Initial Setup

[0772] User: The user installs the "Phone Fraud Prevention App" on their device. After installation is complete, they register the contact information of their relatives and the notification method in case of an emergency on the settings screen. The input data is the contact information of their relatives and the notification method, and the setting information is saved on the device as output.

[0773] Server: The server receives the registration information from the user and performs an authentication process. The input data is the user's registration information, and if the authentication is successful, the output is the registration information stored in a secure database.

[0774] Step 2: Detecting an incoming call

[0775] User: When a call comes in, the user answers the call.

[0776] Terminal: The terminal detects an incoming call and starts capturing the call content in real time when the call starts. The input data is the call start signal, and the output is the start of voice data capture. This operation is stored in buffer memory.

[0777] Step 3: Convert audio data to text

[0778] Terminal (local AI model): The speech recognition engine converts voice data during a call into text data in real time. The input data is voice data, and text data is generated as output. The specific operation is to apply a speech-to-text conversion algorithm.

[0779] Step 4: Fraud Potential Analysis

[0780] Terminal (Local AI Model): The analysis module receives the generated text data and filters out fraudulent words and phrases. The input data is text data, and the output is a determination of whether or not it is likely to be fraudulent. This filtering is performed using keyword search and NLP techniques.

[0781] Step 5: Emotion Recognition

[0782] Terminal (Emotion Engine): Analyzes the user's emotional state based on voice data. The input data is voice and text data, and the output recognizes the user's emotional state (surprise, confusion, anger, etc.). Specific operations include an emotion recognition algorithm that analyzes tone and word choice.

[0783] Step 6: Fraud detection

[0784] Terminal (local AI model): If it determines that there is a high probability of fraud, it activates a fraud alert flag. The input data are the filtering results and emotional state, and the output is a fraud alert flag.

[0785] Step 7: Sending an alert

[0786] Terminal: When the fraud alert flag is activated, a warning message is issued according to the user's emotional state as recognized by the emotion engine. The input data is the fraud alert flag and the emotional state, and the warning message is generated as the output. Specific operations include, for example, outputting a message such as "This call may be fraudulent. Please do not continue the call."

[0787] Step 8: Notify your next of kin

[0788] Terminal: If a fraud is deemed likely, a notification is sent to relatives via email or SMS. The input data is the fraud alert flag and the emotional state, and the output is a notification message. Specific actions include sending a detailed notification such as "The user is very confused. Please contact us immediately."

[0789] Step 9: End the call and save the log

[0790] User: Receives warning and ends call.

[0791] Terminal: After the call ends, the call content and fraud determination results are saved in a secure log file. The input data is the call content and fraud determination results, and log data is generated as output. This log data is stored in secure storage within the terminal. Specifically, the log file records part of the call content (the part suspected of being fraudulent) and details of the fraud determination.

[0792] In this way, the system can effectively protect the elderly from telephone fraud by performing real-time analysis combined with user emotion recognition.

[0793] (Application example 2)

[0794] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0795] This invention relates to a system for preventing elderly people from becoming victims of telephone fraud. In particular, it aims to enhance user safety by not only analyzing the possibility of fraud but also recognizing the user's emotions and responding flexibly according to their state. Current systems only detect fraud and do not take into account the user's emotional state, which often leaves elderly people confused and frightened. For this reason, there is a need for the development of a system that also incorporates emotion recognition.

[0796] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring voice data in real time, means for converting the acquired voice data into text data, means for analyzing the possibility of fraud from the converted text data, means for issuing a warning when there is a possibility of fraud, means for notifying relatives of the possibility of fraud, means for recognizing the user's emotions, means for adjusting the content of the warning depending on the user's emotional state, and means for detailing the content of the notification depending on the emotional state. This reduces the possibility that elderly people will become victims of telephone fraud and enables appropriate responses depending on the user's emotional state.

[0797] "Voice data" refers to data that is recorded or acquired in digital form, such as voice from a conversation.

[0798] "Real-time" means that the processing occurs immediately, without delay.

[0799] "Means" refers to a method or apparatus for achieving a particular function.

[0800] "Text data" is digital data that has been converted from voice data into character information.

[0801] "Fraud likelihood analysis" refers to assessing the likelihood of fraudulent activity based on voice and text data.

[0802] "Issuing a warning" refers to the act of issuing a warning or alert to the user.

[0803] "Relatives" refers to an individual's family or blood relatives, including parents, siblings, and children.

[0804] "Emotion recognition" means analyzing and determining a user's psychological state and emotions (for example, surprise, confusion, anger, etc.) from voice or text.

[0805] "Adjusting the warning content according to the emotional state" means generating a warning message with appropriate expressions and tone, taking into account the emotional state of the user.

[0806] "Detailing the notification content" means making the specific information and content of the notification to relatives more detailed and precise depending on the user's emotional state and the possibility of fraud.

[0807] "Local terminal" refers to a local computer or device that can be directly operated by a user.

[0808] A "local AI model" is an artificial intelligence model that is placed within a specific enclosure or device and operates independently of the internet.

[0809] This invention is a system to protect elderly people from becoming victims of telephone fraud. In particular, it not only analyzes the possibility of fraud but also recognizes the user's emotions and responds flexibly according to their emotional state.

[0810] System configuration

[0811] 1. Real-time acquisition of audio data

[0812] Terminal: Equipped with a function to capture voice data during a call in real time. Specifically, it uses the smartphone's microphone.

[0813] 2. Converting audio data to text data

[0814] On the device: Use the Google Speech-to-Text API to convert captured audio data into text data.

[0815] 3. Analyze for potential fraud

[0816] Terminal: Use spaCy as a natural language processing engine to analyze possible fraud from the converted text data.

[0817] 4. Emotion recognition

[0818] Device: Uses IBM Watson Tone Analyzer to analyze the user's emotional state based on voice and text data.

[0819] 5. Alerts and Notifications

[0820] Terminal: If there is a high possibility of fraud, the system will warn the user and send a notification to their relatives. The warning message will be generated using the results of the emotion engine.

[0821] Server: To notify relatives, we use Twilio to send SMS and SMTP to send email.

[0822] 6. Log storage

[0823] On the device: Call logs and analytics are stored in a SQLite database for secure use.

[0824] Specific examples

[0825] One day, the user receives a call on their smartphone from someone claiming to be from the bank. As soon as the call begins, the device begins capturing the call in real time, and the audio data is converted into text using the Google Speech-to-Text API. If the converted text contains keywords that suggest fraud, such as "account number confirmation required," "unpaid balance," or "transfer," the device analyzes the text data with spaCy and determines that there is a high possibility of fraud. At the same time, the device uses IBM Watson Tone Analyzer to analyze the user's emotional state and recognizes that the user is feeling confused or surprised.

[0826] Based on the results, the device will display a soft warning message to the user, such as "Please stay calm. This call may be fraudulent. Do not continue the call." Additionally, Twilio will be used to send a detailed email notification to the next of kin, such as "The user is very distressed. Please contact us immediately."

[0827] This series of processes is extremely effective in strengthening user security and preventing fraud before it occurs. In addition, the information saved as a log can be used for future verification and as evidence.

[0828] Example prompts for generative AI models

[0829] Write a program that converts voice data into text data, analyzes the text data, and detects possible fraud. This program uses IBM Watson Tone Analyzer to recognize the user's emotions, and if fraud is detected, sends an SMS to relatives using the Twilio API and an email using SMTP. Also, save the call content and analysis results in a SQLite database.

[0830] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0831] Step 1:

[0832] The device acquires voice data in real time. This involves using the smartphone's microphone to capture voice data from the beginning to the end of a call. The input is the voice during the call, and the output is the captured voice data.

[0833] Step 2:

[0834] The device converts the acquired voice data into text data. The Google Speech-to-Text API is used to convert the voice data into text format. The input is voice data and the output is text data. This conversion allows the spoken content to be handled as text information.

[0835] Step 3:

[0836] The device analyzes the converted text data to detect potential fraud. It uses a natural language processing engine (e.g., spaCy) to assess the likelihood of fraud based on specific keywords and phrases. The input is the text data, and the output is a fraud assessment result (whether fraud is likely or unlikely). This analysis identifies indicators of fraudulent activity.

[0837] Step 4:

[0838] An emotion recognition engine is used to analyze the user's emotional state. IBM Watson Tone Analyzer is used to determine the user's emotion (for example, surprise, confusion, anger, etc.) based on text data and voice tone. The input is voice data and text data, and the output is the result of the emotional state determination. This allows the user's psychological state to be understood.

[0839] Step 5:

[0840] If the device is determined to be highly likely to be a fraudster, a warning is issued to the user. Based on the emotion recognition results, a warning message is generated in an appropriate tone. For example, it could say, "This call may be fraudulent. Please do not continue the call." The input is the fraud detection result and the emotional state detection result, and the output is the warning message. This allows the user to be immediately alerted.

[0841] Step 6:

[0842] The device notifies relatives of possible fraud. It uses the Twilio API to send an SMS and an email via SMTP. This notification contains detailed information based on the user's emotional state. For example, it could say, "The user is very confused. Please contact us immediately." The input is the fraud detection result and the emotional state detection result, and the output is the SMS and email sent. This allows relatives to respond quickly.

[0843] Step 7:

[0844] The device saves the call content and analysis results in a secure log file. An SQLite database is used to record the call content, fraud detection results, and emotional state detection results. The input is the text data of the call content, fraud detection results, and emotional state detection results, and the output is the saved log file. This makes it possible to check the call content and fraud detection results later.

[0845] Step 8:

[0846] The user or a relative can check the log file as needed. The log file contains a portion of the call content (parts suspected of fraud) and the analysis results. The input is the request, and the output is the display of the log file. This allows the user to check the details of past call content and the fraud detection status.

[0847] Prompt Sentence Examples

[0848] Write a program that converts voice data into text data, analyzes the text data, and detects possible fraud. This program uses IBM Watson Tone Analyzer to recognize the user's emotions, and if fraud is detected, sends an SMS to relatives using the Twilio API and an email using SMTP. Also, save the call content and analysis results in a SQLite database.

[0849] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0850] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0851] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0852] [Third embodiment]

[0853] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0854] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0855] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0856] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0857] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0858] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0859] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0860] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0861] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0862] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0863] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0864] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0865] This invention is a system for analyzing voice data in real time and providing warnings and notifications, designed to prevent elderly people from becoming victims of telephone fraud. The system of the present invention has the following configuration in particular.

[0866] System configuration

[0867] 1. A means of acquiring audio data in real time

[0868] Device: The user's device is equipped with real-time audio capture capabilities, which allows for instantaneous capture of phone conversations.

[0869] 2. A means of converting the acquired voice data into text data

[0870] Terminal (local AI model): Equipped with a speech recognition engine, it converts call content into text data in real time, allowing voice data to be handled as text information.

[0871] 3. A method for analyzing the possibility of fraud from the converted text data

[0872] Device (Local AI Model): Natural Language Processing (NLP) techniques are used to analyze the converted text data and detect fraudulent words and phrases. The local AI model determines the likelihood of fraud based on known fraud techniques.

[0873] 4. A way to flag potential fraud

[0874] Device: If a potential scam is detected, the device will immediately issue an audio warning to the user, including specific instructions such as "This call may be fraudulent. Do not continue the call."

[0875] 5. How to notify relatives of potential fraud

[0876] Server: If a potential scam is detected, the user's relatives will be notified via email or SMS, with a message such as "The user has received a potentially fraudulent call. Please check."

[0877] Program processing

[0878] Initial Setup

[0879] User: The user installs the "Phone Fraud Prevention App" (hereinafter referred to as the "App") on their device. After installation, they register the contact information of their relatives on a simple setup screen.

[0880] Server: The server receives user registration information and performs authentication and configuration.

[0881] Real-time call analysis

[0882] User: When a user answers the phone, the call begins.

[0883] Terminal: When a call begins, the terminal begins capturing voice data in real time.

[0884] Device (local AI model): The speech recognition engine converts voice data into text in real time.

[0885] On-device (local AI model): Analyzes the converted text data and filters it to see if it contains malicious words or phrases.

[0886] On-device (local AI model): Uses advanced natural language processing capabilities to understand context and assess the likelihood of fraud.

[0887] Fraud detection and notification

[0888] Device (local AI model): Activate fraud alert flag if fraud is determined to be highly likely.

[0889] Device: If the fraud alert flag is activated, the user will hear a voice warning saying, "This call may be fraudulent. Do not continue."

[0890] Device: At the same time, your relatives will be notified by email or SMS of the possible fraud.

[0891] Ending a call and saving a log

[0892] User: The user ends the call.

[0893] Device: After the call is completed, the call details and fraud detection results are saved in a secure log file.

[0894] Device: Saved logs are stored in secure storage on the user's device and can be viewed by the user or their relatives if necessary.

[0895] Specific examples

[0896] One day, the user receives a call on their mobile phone from someone claiming to be from a "public institution." As soon as the call begins, the device begins capturing the call content in real time. The local AI model converts the voice data into text and analyzes the call content. If the call content contains keywords associated with fraud, such as "unpaid bills" or "transfers," the local AI model determines that it is likely a scam. The device then immediately issues a voice warning to the user and simultaneously sends a notification to their relatives. When the call ends, the suspected fraudulent call content and the fraud determination result are recorded in a log and securely stored. This series of processes significantly reduces the risk of elderly people falling victim to fraud.

[0897] The system can provide peace of mind to seniors and their families through real-time voice analysis and immediate response.

[0898] The processing flow will be explained below.

[0899] Step 1:

[0900] User: The user installs the "Phone Fraud Prevention App" on their device. After installation, the initial setup screen is displayed. The user enters the contact information of their relatives and completes the setup.

[0901] Step 2:

[0902] Server: The server receives the user registration information and performs an authentication process. If authentication is successful, the server stores the registration information in a secure database.

[0903] Step 3:

[0904] User: When a call comes in to the user's phone, the user answers the call.

[0905] Step 4:

[0906] Terminal: When a call starts, the terminal starts capturing audio data in real time, which is temporarily stored in the terminal's buffer memory.

[0907] Step 5:

[0908] Device (local AI model): The speech recognition engine converts voice data into text data in real time. This conversion process is performed on the device, and the converted text data is sent to the analysis module.

[0909] Step 6:

[0910] On-device (local AI model): The analysis module receives the converted text data and filters it to analyze potential fraud. If specific keywords or phrases are detected, the content is analyzed in detail.

[0911] Step 7:

[0912] On-device (local AI model): An advanced natural language processing (NLP) engine understands the context and assesses the likelihood of fraud from text data. If fraud is deemed likely, a fraud alert flag is raised.

[0913] Step 8:

[0914] Terminal: When the fraud alert flag is activated, the terminal will immediately issue an audio warning to the user, playing a specific warning message such as "This call may be fraudulent. Do not continue the call."

[0915] Step 9:

[0916] At the same time, the device will send emails and / or SMS to registered contacts of relatives informing them of the potential fraud, including a brief description of the alleged fraud.

[0917] Step 10:

[0918] User: The user receives a warning and ends the call. It is recommended that the call be ended immediately after the warning.

[0919] Step 11:

[0920] Device: After the call ends, the device saves the call details and fraud detection results in a secure log file that is stored in secure storage on the user's device.

[0921] Step 12:

[0922] Device: The user or a relative can review this log file if necessary. The log file contains a portion of the call (suspected fraudulent portion) and details of the fraud determination.

[0923] This series of processing steps results in a system that provides real-time protection for seniors against telephone fraud.

[0924] Example 1

[0925] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0926] As the elderly are increasingly vulnerable to telephone fraud, there is a need to provide a safe calling environment. Conventional countermeasures against telephone fraud are mostly reactive and do not provide real-time fraud prevention measures. This means that the risk of elderly people falling victim to fraud and suffering financial and psychological damage cannot be fully prevented.

[0927] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0928] In this invention, the server includes means for acquiring voice data in real time, means for converting the acquired voice data into text data, means for analyzing the possibility of fraud from the converted text data, means for issuing a warning when there is a possibility of fraud, means for notifying relatives of the possibility of fraud, means for saving data after the call ends, and means for storing the analyzed data in secure storage. This reduces the risk of elderly people falling victim to telephone fraud in real time and enables immediate contact with relatives, thereby providing a safe calling environment.

[0929] "Voice data" means information in analog or digital form that contains the human voice.

[0930] "Real-time" means that processing occurs immediately, without delay.

[0931] A "capturing means" is a method or device for capturing or collecting audio data.

[0932] "Text data" is voice data expressed as text information.

[0933] A "converting means" is a method or device for converting audio data into text data.

[0934] The "analyzing means" refers to a method or device for analyzing the converted text data to detect specific patterns or keywords.

[0935] "Probability of fraud" is the probability that a communication is deemed to contain fraud or fraudulent intent.

[0936] "Warning means" refers to a method or device for sending a sound or message to alert the user.

[0937] "Means of notification" refers to the method or device used to communicate information to relatives.

[0938] "Means for saving data after a call ends" refers to a method or device for saving the contents of the call and the analysis results.

[0939] "Secure storage" refers to storage environments and technologies that ensure data safety.

[0940] "Local terminal" refers to a device or equipment that is directly operated by a user.

[0941] A "local AI model" is an artificial intelligence technology that runs within a device and performs specific analyses and judgments.

[0942] A "means for detecting the start of a call" is a method or device for recognizing that a call has been received or started.

[0943] This invention is a system for preventing elderly people from becoming victims of telephone fraud, and performs real-time analysis of voice data, warnings, and notifications. The detailed configuration and operation of this system will be described below.

[0944] System configuration

[0945] 1. A means of acquiring audio data in real time

[0946] Device: The user's device is equipped with a built-in microphone to capture audio data in real time, in order to detect when a call starts and instantly capture the contents of the call.

[0947] 2. A means of converting the acquired voice data into text data

[0948] Terminal (local AI model): Equipped with a speech recognition engine (e.g., Google Speech-to-Text or Microsoft Azure Cognitive Services), it converts the contents of the call into text data in real time, allowing the voice data to be treated as text information.

[0949] 3. A method for analyzing the possibility of fraud from the converted text data

[0950] Terminal (local AI model): The analysis engine uses natural language processing (NLP) technology to analyze the converted text data, detecting fraudulent words and phrases based on existing fraud techniques and determining the likelihood of fraud.

[0951] 4. A way to flag potential fraud

[0952] Device: If a call is deemed potentially fraudulent, the device will immediately issue an audio warning to the user, including messages such as "This call may be fraudulent. Do not continue the call."

[0953] 5. How to notify relatives of potential fraud

[0954] Server: If a potential scam is detected, the user's relatives will be notified via email and / or SMS, with a message such as "The user has received a potentially fraudulent call. Please check."

[0955] 6. How to save data after a call ends

[0956] On the device: After the call ends, the analyzed text data and the log file containing the fraud detection result are stored in a secure storage location. This storage process is performed in a secure manner so that it can be accessed by the user.

[0957] 7. A means of storing the analyzed data in secure storage

[0958] Device: Saved log files are stored in secure storage on the device and can be viewed by the user or their relatives if necessary.

[0959] Specific examples

[0960] One day, the user receives a call on their mobile phone from someone claiming to be from a public institution. As soon as the call begins, the device begins capturing the call content in real time. The local AI model converts the voice data into text and analyzes the text data. If the call content contains keywords associated with fraud, such as "unpaid bills" or "transfers," the local AI model determines that it is likely a scam. The device then immediately issues an audio warning to the user and simultaneously sends a notification to their relatives. After the call ends, the suspected fraudulent content of the call and the fraud determination result are recorded in a log file and securely stored. This series of processes significantly reduces the risk of elderly people falling victim to fraud.

[0961] Prompt Sentence Examples

[0962] "Please explain the system to be followed when an elderly person receives a fraudulent phone call."

[0963] The system can provide peace of mind to seniors and their families through real-time voice analysis and immediate response.

[0964] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0965] Step 1:

[0966] When a user answers a call, the call begins. The device detects the start of the call and begins capturing audio data using the built-in microphone. The input at this time is the audio signal during the call, and the output is audio data. Specifically, the built-in microphone of the device collects the telephone voice and converts it into digital audio data.

[0967] Step 2:

[0968] The device converts the captured voice data into text data in real time. This uses a local AI model speech recognition engine (e.g., Google Speech-to-Text or Microsoft Azure Cognitive Services). The input is voice data, and the output is text data. Specifically, the speech recognition engine analyzes voice patterns and converts them into corresponding text data.

[0969] Step 3:

[0970] The device analyzes the text data to assess the likelihood of fraud. A local AI model, the Natural Language Processing (NLP) engine, is responsible for this process. The input is the text data, and the output is a flag indicating the likelihood of fraud (a fraud alert flag). Specifically, the NLP engine filters keywords and phrases in the text to detect patterns associated with fraud.

[0971] Step 4:

[0972] When the fraud alert flag is activated, the terminal immediately issues an audio warning to the user. The input is the fraud alert flag and the output is the warning message. Specifically, the terminal plays the audio message "This call may be fraudulent. Do not continue the call."

[0973] Step 5:

[0974] When the fraud alert flag is activated, the server notifies the relative of a possible fraud. The input is a signal sent from the device to the server, and the output is a notification message to the relative. Specifically, the server sends an email or SMS to the relative's registered contacts saying, "A user has answered a suspected fraudulent call. Please check."

[0975] Step 6:

[0976] When the user ends the call, the device saves the call content and the fraud detection result in a secure log file. The input is a call end signal and text data containing the fraud detection result, and the output is a secure log file. Specifically, the device encrypts the data and saves it in secure storage.

[0977] Step 7:

[0978] The saved log file is stored in secure storage within the device. The input is a secure log file, and the output is saved in secure storage. Specifically, the device saves the log file according to a specified security protocol and allows the user or relatives to access it as needed.

[0979] (Application example 1)

[0980] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0981] There is a need to effectively reduce the risk of elderly people becoming victims of telephone fraud. However, it is difficult for elderly people to recognize the signs of fraud themselves, and they are unable to act quickly without appropriate warnings and notifications. To solve these problems, a system that processes voice data in real time and provides immediate warnings is required.

[0982] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0983] In this invention, the server includes a means for acquiring voice data in real time, a means for converting the acquired voice data into text data, a means for analyzing the possibility of fraud from the converted text data, a means for notifying relatives of the possibility of fraud if there is a possibility of fraud, and a means for issuing visual and audio warnings, thereby significantly reducing the risk of elderly people becoming victims of telephone fraud and enabling prompt action.

[0984] "Audio data" refers to recordings of voice during a call, etc., and is information that is subject to signal processing.

[0985] "Means of acquisition in real time" refers to technical means for collecting voice data immediately without delay.

[0986] "Text data" is voice data converted into character information, and is basic data for analysis.

[0987] "Means for converting" refers to the speech recognition technology used to convert voice data into text data.

[0988] "Means for analyzing" refers to algorithms and techniques used to process the captured and converted text data to assess the likelihood of fraud.

[0989] "Alert Measures" refers to the audio and visual interfaces used to notify the user of potential fraud.

[0990] "Means of notifying relatives of possible fraud" refers to notification techniques that use email, SMS, or other means to inform a user's relatives of possible fraud.

[0991] "Visual and audio warning means" refers to devices and technologies that display and transmit visual and audio warnings that are intuitively understandable to the user.

[0992] A "local terminal" is a device distributed to a user, and is an apparatus that has computational resources and storage for real-time processing.

[0993] "Local AI model" refers to an artificial intelligence model that runs within the device and incorporates learning algorithms to detect signs of fraud.

[0994] "Means for detecting the start of a call" refers to a technical means for recognizing that a telephone call has started and immediately starting to capture voice data.

[0995] This invention is a system for analyzing voice data in real time and providing warnings and notifications, designed to prevent elderly people from becoming victims of telephone fraud. The system of the present invention has the following configuration in particular.

[0996] System configuration

[0997] The system includes means for acquiring voice data in real time, means for converting the acquired voice data into text data, means for analyzing the converted text data for the possibility of fraud, means for issuing an alert when there is a possibility of fraud, means for notifying relatives of the possibility of fraud, and means for issuing visual and audio alerts, thereby enabling the user to immediately recognize the risk of fraud and the relatives to quickly respond to the crisis.

[0998] Program processing

[0999] Initial Setup

[1000] User: The user installs the "Phone Fraud Prevention App" on their device. After installation, they register the contact information of their relatives on a simple setup screen.

[1001] Server: The server receives user registration information and performs authentication and configuration.

[1002] Real-time call analysis

[1003] Terminal (smart glasses): When a call starts, the microphone built into the smart glasses captures voice data in real time.

[1004] Terminal: The acquired voice data is converted into text data using a voice recognition engine (e.g., Google API).

[1005] On-device (local AI model): Analyzes the converted text data to see if it contains fraudulent words or phrases. This uses the ScamPredictionModel.

[1006] Fraud detection and notification

[1007] Device (local AI model): Activate fraud alert flag if fraud is determined to be highly likely.

[1008] Device: If the fraud alert flag is activated, the user will be warned audibly, "This call may be fraudulent. Do not continue the call." A visual warning will also be displayed on the smart glasses display.

[1009] Server: At the same time, notify the relatives of the possible fraud via email or SMS (using the smtplib library).

[1010] Ending a call and saving a log

[1011] User: The user ends the call.

[1012] Device: After the call is completed, the call details and fraud detection results are saved in a secure log file.

[1013] On the device: The saved logs are stored in secure storage and can be viewed by the user or a relative if necessary.

[1014] Specific examples

[1015] One day, the user receives a call through their smart glasses from someone claiming to be from a government agency. As soon as the call begins, the smart glasses begin capturing the call content in real time. The local AI model converts the audio data into text and analyzes the call content. If the call content contains keywords associated with fraud, such as "unpaid bills" or "transfers," the local AI model determines that it is likely a scam. The smart glasses then immediately issue an audio and visual warning to the user and simultaneously send an emergency notification to their relatives. When the call ends, the suspected fraudulent call content and the fraud determination result are logged and securely stored.

[1016] Generative AI model prompt example

[1017] "Develop an AI model to identify potentially fraudulent phone call content. The model will take Japanese text data as input and determine whether it is likely to be fraudulent. Examples of phrases include: "unpaid money", "transfer". The output should be a Boolean value indicating the likelihood of fraud."

[1018] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1019] Step 1:

[1020] User initiates a call

[1021] Input: A user receives a call through their smart glasses.

[1022] How it works: The smart glasses' call start detection sensor recognizes the start of a call and starts collecting voice data.

[1023] Output: A call initiation signal is sent to the processing unit of the smart glasses.

[1024] Step 2:

[1025] Acquiring audio data

[1026] Input: Call initiation signal and call audio.

[1027] How it works: The smart glasses have a built-in microphone that captures audio data in real time, which is then converted into digital form and sent to a processing unit.

[1028] Output: Digital signal of audio data.

[1029] Step 3:

[1030] Converting audio data to text

[1031] Input: Digital signal of audio data.

[1032] How it works: Using a speech recognition engine (e.g. Google API) within the smart glasses, the voice data is converted into text data. During this conversion process, the voice signal is analyzed and converted into corresponding text information.

[1033] Output: Text data.

[1034] Step 4:

[1035] Text data analysis

[1036] Input: Text data.

[1037] How it works: A local AI model (ScamPredictionModel) is used to analyze text data. The analysis process detects keywords and context within the text data and assesses its likelihood of fraud.

[1038] Output: A flag indicating whether fraud is likely.

[1039] Step 5:

[1040] Fraud Alert Verification

[1041] Input: Possible fraud flag.

[1042] How it works: If the fraud alert flag is active, the system determines that fraud is likely.

[1043] Output: Alert signal.

[1044] Step 6:

[1045] Sending a warning to the user

[1046] Input: Alert signal.

[1047] How it works: The smart glasses provide users with visual and audio warnings, including a message on the display that reads "This call may be fraudulent. Do not continue the call," as well as an audio warning.

[1048] Output: A warning notice to the user.

[1049] Step 7:

[1050] Sending notifications to relatives

[1051] Input: Alert signal.

[1052] What it does: The server sends emergency notifications via email or SMS based on the contact information of the next of kin configured. It uses the smtplib library to send the notifications.

[1053] Output: Notification sent to relatives.

[1054] Step 8:

[1055] Call content and result logging

[1056] Input: call termination signal, call content, fraud detection result.

[1057] How it works: When a user ends a call, the device saves the call and the fraud detection result in a secure log file that is encrypted and stored in secure storage on the user's device.

[1058] Output: Secure log file.

[1059] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1060] This invention is a system that performs real-time analysis of voice data and provides warnings and notifications to protect elderly people from becoming victims of telephone fraud. In particular, by combining this system with an emotion engine that recognizes the user's emotions, this system enables flexible responses according to the user's emotional state.

[1061] System configuration

[1062] 1. A means of acquiring audio data in real time

[1063] Device: The user's device is equipped with the capability to capture phone conversations in real time.

[1064] 2. A means of converting the acquired voice data into text data

[1065] Device (local AI model): A speech recognition engine converts voice data into text data in real time. This conversion process is performed within the device.

[1066] 3. A method for analyzing the possibility of fraud from the converted text data

[1067] On-device (local AI model): The analytics module analyzes the converted text data and filters it to detect fraudulent words and phrases. It uses natural language processing (NLP) techniques to understand the context and assess the likelihood of fraud.

[1068] 4. A way to flag potential fraud

[1069] Terminal: If a call is deemed potentially fraudulent, the terminal will issue an audio warning to the user, such as a warning message saying, "This call may be fraudulent. Do not continue the call."

[1070] 5. How to notify relatives of potential fraud

[1071] Server: Notify the user's relatives of the potential scam via email or SMS. This notification may include a message such as "The user has received a potentially fraudulent call. Please check."

[1072] 6. Means of recognizing user emotions using an emotion engine

[1073] The device is equipped with an emotion engine that analyzes the user's emotional state based on voice data. The emotion engine analyzes the tone of voice and word choice to gauge the user's emotions (e.g., surprise, confusion, anger, etc.).

[1074] 7. A means for the emotion engine to tailor warning content based on the user's emotional state

[1075] Terminal: The emotion engine recognizes the user's emotional state and issues a warning message according to the user's emotion. For example, if the user is surprised, it will provide a gentle warning message such as "Please stay calm. This call may be a scam."

[1076] 8. A method for detailing notifications to relatives based on the user's emotional state

[1077] Device: If the user's emotional state exceeds a certain threshold, the notification to the next of kin will be detailed. For example, if the user is very distressed, the next of kin will receive a detailed message saying, "The user is very distressed. Please contact us immediately."

[1078] Program processing

[1079] Initial Setup

[1080] User: The user installs the "Phone Fraud Prevention App" on their device. After installation, they register the contact information of their relatives and the method of notification in case of an emergency on the settings screen.

[1081] Server: The server receives the user registration information, performs an authentication process, and, if successful, stores the registration information in a secure database.

[1082] Real-time call analysis and emotion recognition

[1083] User: When a call comes in to the user's phone, answer the call.

[1084] Terminal: When a call starts, the terminal starts capturing audio data in real time, which is temporarily stored in the terminal's buffer memory.

[1085] Device (local AI model): The speech recognition engine converts voice data into text data in real time.

[1086] Terminal (local AI model): The analysis module receives the converted text data and filters it to analyze possible fraud.

[1087] On-device (local AI model): A natural language processing engine understands the context and assesses the likelihood of fraud.

[1088] Terminal (emotion engine): Analyzes the user's emotional state based on voice data and recognizes emotions such as surprise, confusion, or anger.

[1089] Fraud detection and emotion-based warnings and notifications

[1090] Device (local AI model): Activate fraud alert flag if fraud is determined to be highly likely.

[1091] Terminal: When the fraud alert flag is activated, the emotion engine recognizes the user's emotional state and issues a warning message, such as "This call may be fraudulent. Do not continue the call."

[1092] On the device: At the same time, an email or SMS is sent to a relative notifying them of the possible scam. If the user's emotional state is very distressed, the notification will include a detailed message such as "The user is very distressed. Please contact us immediately."

[1093] Ending a call and saving a log

[1094] User: The user receives an alert and ends the call.

[1095] Device: After the call is completed, the call details and fraud detection results are saved in a secure log file, which is stored in secure storage on the user's device.

[1096] On the device: If necessary, the user or a relative can review the log file, which contains a portion of the call (where fraud is suspected) and details of the fraud determination.

[1097] Specific examples

[1098] One day, a user receives a call from someone claiming to be from a bank. As soon as the call begins, the device begins capturing the conversation in real time. A local AI model converts the voice data into text and analyzes the conversation. If the call contains keywords associated with fraud, such as "account number confirmation required," "unpaid bill," or "transfer," the emotion engine analyzes the user's tone of voice and word choice and recognizes that the user is confused. In this case, the device issues a warning message to the user: "Please remain calm. This call may be fraudulent. Do not continue the call." At the same time, a detailed email is sent to the user's relatives, stating, "The user is very confused. Please contact us immediately." After the user ends the call, the conversation and the fraud detection result are securely stored as a log. This process provides better protection for the elderly against telephone fraud.

[1099] The system performs real-time voice analysis as well as user emotion recognition, detecting fraud and taking appropriate action to provide peace of mind to seniors and their families.

[1100] The processing flow will be explained below.

[1101] Step 1:

[1102] User: The user installs the "Phone Fraud Prevention App" on their device. After installation, they launch the app and register on the initial setup screen. The user enters contact information for relatives and emergency notification methods to complete the setup.

[1103] Step 2:

[1104] Server: The server receives the user registration information and performs an authentication process. If authentication is successful, the user registration information is stored in a secure database.

[1105] Step 3:

[1106] User: When a call comes in to the user's phone, the user answers the call.

[1107] Step 4:

[1108] Terminal: When a call starts, the terminal starts capturing audio data in real time, which is temporarily stored in the terminal's buffer memory.

[1109] Step 5:

[1110] Device (local AI model): The speech recognition engine converts voice data into text data in real time. This conversion process is performed on the device, and the converted text data is sent to the analysis module.

[1111] Step 6:

[1112] On-device (local AI model): The analytics module receives the text data and filters it to determine whether it is potentially fraudulent, checking to see if specific fraud-related keywords or phrases are detected.

[1113] Step 7:

[1114] On-device (local AI model): Utilizing advanced natural language processing (NLP) to understand context and perform a detailed assessment of the likelihood of fraud. If fraud is deemed likely, a fraud alert flag is raised.

[1115] Step 8:

[1116] Device (Emotion Engine): Analyzes the user's emotional state using voice data. The emotion engine recognizes the user's emotional state (surprise, confusion, anger, etc.) based on the user's tone of voice and word choice.

[1117] Step 9:

[1118] Terminal: If the emotion engine determines that the user's emotion is negative, such as surprise or confusion, it generates a gentle warning message, such as "Please stay calm. This call may be fraudulent. Do not continue the call."

[1119] Step 10:

[1120] Terminal: If the fraud alert flag is activated, the terminal will audibly reproduce the generated warning message to the user.

[1121] Step 11:

[1122] Device: At the same time, promptly notify relatives, for example by sending an email or SMS with detailed information such as "User is receiving a scam call. This is particularly disturbing. Please contact them immediately," depending on the user's emotional state.

[1123] Step 12:

[1124] User: The user is warned and ends the call. Users who hear the warning are encouraged to end the call immediately.

[1125] Step 13:

[1126] Device: After the call is completed, the call details and fraud detection results are saved to a secure log file, which is stored in secure storage on the user's device.

[1127] Step 14:

[1128] Device: The user or a relative can check this log file as needed. The log file contains detailed records of suspected fraudulent calls and their evaluation results.

[1129] This provides real-time protection for seniors against phone scams, allows for flexible responses based on emotional state, and facilitates rapid response in emergencies by notifying relatives.

[1130] Example 2

[1131] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1132] To prevent elderly people from becoming victims of telephone fraud, a system that can quickly detect potential fraud and issue a warning is essential. However, conventional systems issue a uniform warning message without taking into account the user's emotional state, which can lead to users being unable to respond appropriately. Furthermore, notifications to relatives are also sent in a uniform manner, which creates an issue where the urgency of the situation is not accurately conveyed.

[1133] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring voice data in real time, means for converting the acquired voice data into text data, means for analyzing the possibility of fraud from the converted text data, means for issuing a warning when there is a possibility of fraud, means for recognizing the subscriber's emotion, means for adjusting the content of the warning depending on the recognized emotional state, and means for notifying relatives of the possibility of fraud. This enables flexible response according to the user's emotional state, effectively protecting the elderly from telephone fraud, and enabling accurate notification to relatives.

[1134] "Audio data" is data that represents an audio signal in digital form and is used to record the contents of a call.

[1135] "Real-time" refers to processing that occurs immediately without delay, meaning that voice data is acquired and processed immediately during a call.

[1136] "Text data" is data that expresses character information in digital format, and is obtained by converting voice data into a string of characters.

[1137] "Fraud likelihood" refers to assessing the likelihood that a call contains fraudulent activity and determining whether the risk is high or low.

[1138] A "warning" is a message provided by voice, text, or other means to alert the user when there is a high possibility of fraud.

[1139] "Subscribers" refer to individuals who use the system, which is primarily targeted at elderly people.

[1140] "Emotions" refer to an individual's psychological state or feelings, and specifically include states such as surprise, confusion, and anger.

[1141] "Emotional state" indicates the user's current emotional state, as analyzed by the emotion engine.

[1142] "Relatives" are family members or close relatives of the user, who are to be notified in the event of an emergency.

[1143] "Notification" is a means of conveying important information to others, and is sent to relatives in the form of email, SMS, etc.

[1144] MODE FOR CARRYING OUT THE INVENTION

[1145] System Overview

[1146] This invention is a system that performs real-time analysis of voice data and provides warnings and notifications to protect elderly people from becoming victims of telephone fraud. By combining this system with an emotion engine that recognizes the user's emotions, it enables flexible responses according to the user's emotional state.

[1147] Hardware and Software Configuration

[1148] 1. A means of acquiring audio data in real time

[1149] Device: The user's device is equipped with the capability to capture the contents of phone conversations in real time.

[1150] 2. A means of converting the acquired voice data into text data

[1151] Device (local AI model): The speech recognition engine converts voice data into text data in real time. This conversion process is performed within the device.

[1152] 3. A method for analyzing the possibility of fraud from the converted text data

[1153] On-device (local AI model): The analysis module analyzes the converted text data and filters it to detect fraudulent words and phrases. It uses natural language processing (NLP) techniques to understand the context and assess the likelihood of fraud.

[1154] 4. A way to flag potential fraud

[1155] Terminal: If a fraudulent call is detected, the terminal will issue an audio warning to the user, such as a warning message saying, "This call may be fraudulent. Do not continue the call."

[1156] 5. How to notify relatives of potential fraud

[1157] Server: The user's relatives are notified by email or SMS of a possible scam. This notification may include a message such as "The user has received a potentially scam call. Please check."

[1158] 6. Means of recognizing user emotions using an emotion engine

[1159] The device is equipped with an emotion engine that analyzes the user's emotional state based on voice data. The emotion engine analyzes the tone of voice and word choice to gauge the user's emotions (e.g., surprise, confusion, anger, etc.).

[1160] 7. A means for the emotion engine to tailor warning content based on the user's emotional state

[1161] Terminal: The emotion engine recognizes the user's emotional state and issues a warning message according to the user's emotion. For example, if the user is surprised, it will provide a gentle warning message such as "Please stay calm. This call may be a scam."

[1162] 8. A method for detailing notifications to relatives based on the user's emotional state

[1163] Device: If the user's emotional state exceeds a certain threshold, the notification to the relatives will be detailed. For example, if the user is very distressed, the relatives will receive a detailed message such as "The user is very distressed. Please contact us immediately."

[1164] Program processing explanation

[1165] Initial Setup

[1166] User: Install the "Phone Fraud Prevention App" on the device. After installation, register the contact information of relatives and the notification method in case of an emergency on the settings screen.

[1167] Server: Receives the user's registration information, performs the authentication process, and, if successful, stores the registration information in a secure database.

[1168] Real-time call analysis and emotion recognition

[1169] User: When a call comes in to the user's phone, answer the call.

[1170] Terminal: When a call starts, the terminal starts capturing voice data in real time, which is temporarily stored in the terminal's buffer memory.

[1171] Device (local AI model): The voice recognition engine converts voice data into text data in real time.

[1172] Terminal (local AI model): The analysis module receives the converted text data and filters it to analyze the possibility of fraud.

[1173] On-device (local AI model): The NLP engine understands the context and assesses the likelihood of fraud.

[1174] Terminal (emotion engine): Analyzes the user's emotional state based on voice data and recognizes emotions such as surprise, confusion, or anger.

[1175] Specific examples

[1176] Specific operation scenario

[1177] One day, the user receives a call from someone claiming to be from the bank. As soon as the call begins, the device begins capturing the conversation in real time. A local AI model converts the voice data into text and analyzes the conversation. If the call contains keywords associated with fraud, such as "account number confirmation required," "unpaid bill," or "transfer," the emotion engine analyzes the user's tone of voice and word choice and recognizes that the user is confused. In this case, the device issues a warning message to the user: "Please remain calm. This call may be fraudulent. Do not continue the call." At the same time, a detailed email is sent to the user's relatives stating, "The user is very confused. Please contact us immediately." After the user ends the call, the conversation and the fraud determination results are securely stored as logs.

[1178] Example prompts for generative AI models

[1179] "Generate natural language explanations for a system that utilizes an emotion engine to detect elderly phone scams in real time and issue appropriate warnings and notifications. The system analyzes voice data, recognizes emotions, alerts, and notifies."

[1180] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1181] Step 1: Initial Setup

[1182] User: The user installs the "Phone Fraud Prevention App" on their device. After installation is complete, they register the contact information of their relatives and the notification method in case of an emergency on the settings screen. The input data is the contact information of their relatives and the notification method, and the setting information is saved on the device as output.

[1183] Server: The server receives the registration information from the user and performs an authentication process. The input data is the user's registration information, and if the authentication is successful, the output is the registration information stored in a secure database.

[1184] Step 2: Detecting an incoming call

[1185] User: When a call comes in, the user answers the call.

[1186] Terminal: The terminal detects an incoming call and starts capturing the call content in real time when the call starts. The input data is the call start signal, and the output is the start of voice data capture. This operation is stored in buffer memory.

[1187] Step 3: Convert audio data to text

[1188] Terminal (local AI model): The speech recognition engine converts voice data during a call into text data in real time. The input data is voice data, and text data is generated as output. The specific operation is to apply a speech-to-text conversion algorithm.

[1189] Step 4: Fraud Potential Analysis

[1190] Terminal (Local AI Model): The analysis module receives the generated text data and filters out fraudulent words and phrases. The input data is text data, and the output is a determination of whether or not it is likely to be fraudulent. This filtering is performed using keyword search and NLP techniques.

[1191] Step 5: Emotion Recognition

[1192] Terminal (Emotion Engine): Analyzes the user's emotional state based on voice data. The input data is voice and text data, and the output recognizes the user's emotional state (surprise, confusion, anger, etc.). Specific operations include an emotion recognition algorithm that analyzes tone and word choice.

[1193] Step 6: Fraud detection

[1194] Terminal (local AI model): If it determines that there is a high probability of fraud, it activates a fraud alert flag. The input data are the filtering results and emotional state, and the output is a fraud alert flag.

[1195] Step 7: Sending an alert

[1196] Terminal: When the fraud alert flag is activated, a warning message is issued according to the user's emotional state as recognized by the emotion engine. The input data is the fraud alert flag and the emotional state, and the warning message is generated as the output. Specific operations include, for example, outputting a message such as "This call may be fraudulent. Please do not continue the call."

[1197] Step 8: Notify your next of kin

[1198] Terminal: If a fraud is deemed likely, a notification is sent to relatives via email or SMS. The input data is the fraud alert flag and the emotional state, and the output is a notification message. Specific actions include sending a detailed notification such as "The user is very confused. Please contact us immediately."

[1199] Step 9: End the call and save the log

[1200] User: Receives warning and ends call.

[1201] Terminal: After the call ends, the call content and fraud determination results are saved in a secure log file. The input data is the call content and fraud determination results, and log data is generated as output. This log data is stored in secure storage within the terminal. Specifically, the log file records part of the call content (the part suspected of being fraudulent) and details of the fraud determination.

[1202] In this way, the system can effectively protect the elderly from telephone fraud by performing real-time analysis combined with user emotion recognition.

[1203] (Application example 2)

[1204] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1205] This invention relates to a system for preventing elderly people from becoming victims of telephone fraud. In particular, it aims to enhance user safety by not only analyzing the possibility of fraud but also recognizing the user's emotions and responding flexibly according to their state. Current systems only detect fraud and do not take into account the user's emotional state, which often leaves elderly people confused and frightened. For this reason, there is a need for the development of a system that also incorporates emotion recognition.

[1206] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring voice data in real time, means for converting the acquired voice data into text data, means for analyzing the possibility of fraud from the converted text data, means for issuing a warning when there is a possibility of fraud, means for notifying relatives of the possibility of fraud, means for recognizing the user's emotions, means for adjusting the content of the warning depending on the user's emotional state, and means for detailing the content of the notification depending on the emotional state. This reduces the possibility that elderly people will become victims of telephone fraud and enables appropriate responses depending on the user's emotional state.

[1207] "Voice data" refers to data that is recorded or acquired in digital form, such as voice from a conversation.

[1208] "Real-time" means that the processing occurs immediately, without delay.

[1209] "Means" refers to a method or apparatus for achieving a particular function.

[1210] "Text data" is digital data that has been converted from voice data into character information.

[1211] "Fraud likelihood analysis" refers to assessing the likelihood of fraudulent activity based on voice and text data.

[1212] "Issuing a warning" refers to the act of issuing a warning or alert to the user.

[1213] "Relatives" refers to an individual's family or blood relatives, including parents, siblings, and children.

[1214] "Emotion recognition" means analyzing and determining a user's psychological state and emotions (for example, surprise, confusion, anger, etc.) from voice or text.

[1215] "Adjusting the warning content according to the emotional state" means generating a warning message with appropriate expressions and tone, taking into account the emotional state of the user.

[1216] "Detailing the notification content" means making the specific information and content of the notification to relatives more detailed and precise depending on the user's emotional state and the possibility of fraud.

[1217] "Local terminal" refers to a local computer or device that can be directly operated by a user.

[1218] A "local AI model" is an artificial intelligence model that is placed within a specific enclosure or device and operates independently of the internet.

[1219] This invention is a system to protect elderly people from becoming victims of telephone fraud. In particular, it not only analyzes the possibility of fraud but also recognizes the user's emotions and responds flexibly according to their emotional state.

[1220] System configuration

[1221] 1. Real-time acquisition of audio data

[1222] Terminal: Equipped with a function to capture voice data during a call in real time. Specifically, it uses the smartphone's microphone.

[1223] 2. Converting audio data to text data

[1224] On the device: Use the Google Speech-to-Text API to convert captured audio data into text data.

[1225] 3. Analyze for potential fraud

[1226] Terminal: Use spaCy as a natural language processing engine to analyze possible fraud from the converted text data.

[1227] 4. Emotion recognition

[1228] Device: Uses IBM Watson Tone Analyzer to analyze the user's emotional state based on voice and text data.

[1229] 5. Alerts and Notifications

[1230] Terminal: If there is a high possibility of fraud, the system will warn the user and send a notification to their relatives. The warning message will be generated using the results of the emotion engine.

[1231] Server: To notify relatives, we use Twilio to send SMS and SMTP to send email.

[1232] 6. Log storage

[1233] On the device: Call logs and analytics are stored in a SQLite database for secure use.

[1234] Specific examples

[1235] One day, the user receives a call on their smartphone from someone claiming to be from the bank. As soon as the call begins, the device begins capturing the call in real time, and the audio data is converted into text using the Google Speech-to-Text API. If the converted text contains keywords that suggest fraud, such as "account number confirmation required," "unpaid balance," or "transfer," the device analyzes the text data with spaCy and determines that there is a high possibility of fraud. At the same time, the device uses IBM Watson Tone Analyzer to analyze the user's emotional state and recognizes that the user is feeling confused or surprised.

[1236] Based on the results, the device will display a soft warning message to the user, such as "Please stay calm. This call may be fraudulent. Do not continue the call." Additionally, Twilio will be used to send a detailed email notification to the next of kin, such as "The user is very distressed. Please contact us immediately."

[1237] This series of processes is extremely effective in strengthening user security and preventing fraud before it occurs. In addition, the information saved as a log can be used for future verification and as evidence.

[1238] Example prompts for generative AI models

[1239] Write a program that converts voice data into text data, analyzes the text data, and detects possible fraud. This program uses IBM Watson Tone Analyzer to recognize the user's emotions, and if fraud is detected, sends an SMS to relatives using the Twilio API and an email using SMTP. Also, save the call content and analysis results in a SQLite database.

[1240] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1241] Step 1:

[1242] The device acquires voice data in real time. This involves using the smartphone's microphone to capture voice data from the beginning to the end of a call. The input is the voice during the call, and the output is the captured voice data.

[1243] Step 2:

[1244] The device converts the acquired voice data into text data. The Google Speech-to-Text API is used to convert the voice data into text format. The input is voice data and the output is text data. This conversion allows the spoken content to be handled as text information.

[1245] Step 3:

[1246] The device analyzes the converted text data to detect potential fraud. It uses a natural language processing engine (e.g., spaCy) to assess the likelihood of fraud based on specific keywords and phrases. The input is the text data, and the output is a fraud assessment result (whether fraud is likely or unlikely). This analysis identifies indicators of fraudulent activity.

[1247] Step 4:

[1248] An emotion recognition engine is used to analyze the user's emotional state. IBM Watson Tone Analyzer is used to determine the user's emotion (for example, surprise, confusion, anger, etc.) based on text data and voice tone. The input is voice data and text data, and the output is the result of the emotional state determination. This allows the user's psychological state to be understood.

[1249] Step 5:

[1250] If the device is determined to be highly likely to be a fraudster, a warning is issued to the user. Based on the emotion recognition results, a warning message is generated in an appropriate tone. For example, it could say, "This call may be fraudulent. Please do not continue the call." The input is the fraud detection result and the emotional state detection result, and the output is the warning message. This allows the user to be immediately alerted.

[1251] Step 6:

[1252] The device notifies relatives of possible fraud. It uses the Twilio API to send an SMS and an email via SMTP. This notification contains detailed information based on the user's emotional state. For example, it could say, "The user is very confused. Please contact us immediately." The input is the fraud detection result and the emotional state detection result, and the output is the SMS and email sent. This allows relatives to respond quickly.

[1253] Step 7:

[1254] The device saves the call content and analysis results in a secure log file. An SQLite database is used to record the call content, fraud detection results, and emotional state detection results. The input is the text data of the call content, fraud detection results, and emotional state detection results, and the output is the saved log file. This makes it possible to check the call content and fraud detection results later.

[1255] Step 8:

[1256] The user or a relative can check the log file as needed. The log file contains a portion of the call content (parts suspected of fraud) and the analysis results. The input is the request, and the output is the display of the log file. This allows the user to check the details of past call content and the fraud detection status.

[1257] Prompt Sentence Examples

[1258] Write a program that converts voice data into text data, analyzes the text data, and detects possible fraud. This program uses IBM Watson Tone Analyzer to recognize the user's emotions, and if fraud is detected, sends an SMS to relatives using the Twilio API and an email using SMTP. Also, save the call content and analysis results in a SQLite database.

[1259] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1260] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1261] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1262] [Fourth embodiment]

[1263] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1264] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1265] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1266] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1267] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1268] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1269] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1270] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1271] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1272] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1273] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1274] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1275] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1276] This invention is a system for analyzing voice data in real time and providing warnings and notifications, designed to prevent elderly people from becoming victims of telephone fraud. The system of the present invention has the following configuration in particular.

[1277] System configuration

[1278] 1. A means of acquiring audio data in real time

[1279] Device: The user's device is equipped with real-time audio capture capabilities, which allows for instantaneous capture of phone conversations.

[1280] 2. A means of converting the acquired voice data into text data

[1281] Terminal (local AI model): Equipped with a speech recognition engine, it converts call content into text data in real time, allowing voice data to be handled as text information.

[1282] 3. A method for analyzing the possibility of fraud from the converted text data

[1283] Device (Local AI Model): Natural Language Processing (NLP) techniques are used to analyze the converted text data and detect fraudulent words and phrases. The local AI model determines the likelihood of fraud based on known fraud techniques.

[1284] 4. A way to flag potential fraud

[1285] Device: If a potential scam is detected, the device will immediately issue an audio warning to the user, including specific instructions such as "This call may be fraudulent. Do not continue the call."

[1286] 5. How to notify relatives of potential fraud

[1287] Server: If a potential scam is detected, the user's relatives will be notified via email or SMS, with a message such as "The user has received a potentially fraudulent call. Please check."

[1288] Program processing

[1289] Initial Setup

[1290] User: The user installs the "Phone Fraud Prevention App" (hereinafter referred to as the "App") on their device. After installation, they register the contact information of their relatives on a simple setup screen.

[1291] Server: The server receives user registration information and performs authentication and configuration.

[1292] Real-time call analysis

[1293] User: When a user answers the phone, the call begins.

[1294] Terminal: When a call begins, the terminal begins capturing voice data in real time.

[1295] Device (local AI model): The speech recognition engine converts voice data into text in real time.

[1296] On-device (local AI model): Analyzes the converted text data and filters it to see if it contains malicious words or phrases.

[1297] On-device (local AI model): Uses advanced natural language processing capabilities to understand context and assess the likelihood of fraud.

[1298] Fraud detection and notification

[1299] Device (local AI model): Activate fraud alert flag if fraud is determined to be highly likely.

[1300] Device: If the fraud alert flag is activated, the user will hear a voice warning saying, "This call may be fraudulent. Do not continue."

[1301] Device: At the same time, your relatives will be notified by email or SMS of the possible fraud.

[1302] Ending a call and saving a log

[1303] User: The user ends the call.

[1304] Device: After the call is completed, the call details and fraud detection results are saved in a secure log file.

[1305] Device: Saved logs are stored in secure storage on the user's device and can be viewed by the user or their relatives if necessary.

[1306] Specific examples

[1307] One day, the user receives a call on their mobile phone from someone claiming to be from a "public institution." As soon as the call begins, the device begins capturing the call content in real time. The local AI model converts the voice data into text and analyzes the call content. If the call content contains keywords associated with fraud, such as "unpaid bills" or "transfers," the local AI model determines that it is likely a scam. The device then immediately issues a voice warning to the user and simultaneously sends a notification to their relatives. When the call ends, the suspected fraudulent call content and the fraud determination result are recorded in a log and securely stored. This series of processes significantly reduces the risk of elderly people falling victim to fraud.

[1308] The system can provide peace of mind to seniors and their families through real-time voice analysis and immediate response.

[1309] The processing flow will be explained below.

[1310] Step 1:

[1311] User: The user installs the "Phone Fraud Prevention App" on their device. After installation, the initial setup screen is displayed. The user enters the contact information of their relatives and completes the setup.

[1312] Step 2:

[1313] Server: The server receives the user registration information and performs an authentication process. If authentication is successful, the server stores the registration information in a secure database.

[1314] Step 3:

[1315] User: When a call comes in to the user's phone, the user answers the call.

[1316] Step 4:

[1317] Terminal: When a call starts, the terminal starts capturing audio data in real time, which is temporarily stored in the terminal's buffer memory.

[1318] Step 5:

[1319] Device (local AI model): The speech recognition engine converts voice data into text data in real time. This conversion process is performed on the device, and the converted text data is sent to the analysis module.

[1320] Step 6:

[1321] On-device (local AI model): The analysis module receives the converted text data and filters it to analyze potential fraud. If specific keywords or phrases are detected, the content is analyzed in detail.

[1322] Step 7:

[1323] On-device (local AI model): An advanced natural language processing (NLP) engine understands the context and assesses the likelihood of fraud from text data. If fraud is deemed likely, a fraud alert flag is raised.

[1324] Step 8:

[1325] Terminal: When the fraud alert flag is activated, the terminal will immediately issue an audio warning to the user, playing a specific warning message such as "This call may be fraudulent. Do not continue the call."

[1326] Step 9:

[1327] At the same time, the device will send emails and / or SMS to registered contacts of relatives informing them of the potential fraud, including a brief description of the alleged fraud.

[1328] Step 10:

[1329] User: The user receives a warning and ends the call. It is recommended that the call be ended immediately after the warning.

[1330] Step 11:

[1331] Device: After the call ends, the device saves the call details and fraud detection results in a secure log file that is stored in secure storage on the user's device.

[1332] Step 12:

[1333] Device: The user or a relative can review this log file if necessary. The log file contains a portion of the call (suspected fraudulent portion) and details of the fraud determination.

[1334] This series of processing steps results in a system that provides real-time protection for seniors against telephone fraud.

[1335] Example 1

[1336] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1337] As the elderly are increasingly vulnerable to telephone fraud, there is a need to provide a safe calling environment. Conventional countermeasures against telephone fraud are mostly reactive and do not provide real-time fraud prevention measures. This means that the risk of elderly people falling victim to fraud and suffering financial and psychological damage cannot be fully prevented.

[1338] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1339] In this invention, the server includes means for acquiring voice data in real time, means for converting the acquired voice data into text data, means for analyzing the possibility of fraud from the converted text data, means for issuing a warning when there is a possibility of fraud, means for notifying relatives of the possibility of fraud, means for saving data after the call ends, and means for storing the analyzed data in secure storage. This reduces the risk of elderly people falling victim to telephone fraud in real time and enables immediate contact with relatives, thereby providing a safe calling environment.

[1340] "Voice data" means information in analog or digital form that contains the human voice.

[1341] "Real-time" means that processing occurs immediately, without delay.

[1342] A "capturing means" is a method or device for capturing or collecting audio data.

[1343] "Text data" is voice data expressed as text information.

[1344] A "converting means" is a method or device for converting audio data into text data.

[1345] The "analyzing means" refers to a method or device for analyzing the converted text data to detect specific patterns or keywords.

[1346] "Probability of fraud" is the probability that a communication is deemed to contain fraud or fraudulent intent.

[1347] "Warning means" refers to a method or device for sending a sound or message to alert the user.

[1348] "Means of notification" refers to the method or device used to communicate information to relatives.

[1349] "Means for saving data after a call ends" refers to a method or device for saving the contents of the call and the analysis results.

[1350] "Secure storage" refers to storage environments and technologies that ensure data safety.

[1351] "Local terminal" refers to a device or equipment that is directly operated by a user.

[1352] A "local AI model" is an artificial intelligence technology that runs within a device and performs specific analyses and judgments.

[1353] A "means for detecting the start of a call" is a method or device for recognizing that a call has been received or started.

[1354] This invention is a system for preventing elderly people from becoming victims of telephone fraud, and performs real-time analysis of voice data, warnings, and notifications. The detailed configuration and operation of this system will be described below.

[1355] System configuration

[1356] 1. A means of acquiring audio data in real time

[1357] Device: The user's device is equipped with a built-in microphone to capture audio data in real time, in order to detect when a call starts and instantly capture the contents of the call.

[1358] 2. A means of converting the acquired voice data into text data

[1359] Terminal (local AI model): Equipped with a speech recognition engine (e.g., Google Speech-to-Text or Microsoft Azure Cognitive Services), it converts the contents of the call into text data in real time, allowing the voice data to be treated as text information.

[1360] 3. A method for analyzing the possibility of fraud from the converted text data

[1361] Terminal (local AI model): The analysis engine uses natural language processing (NLP) technology to analyze the converted text data, detecting fraudulent words and phrases based on existing fraud techniques and determining the likelihood of fraud.

[1362] 4. A way to flag potential fraud

[1363] Device: If a call is deemed potentially fraudulent, the device will immediately issue an audio warning to the user, including messages such as "This call may be fraudulent. Do not continue the call."

[1364] 5. How to notify relatives of potential fraud

[1365] Server: If a potential scam is detected, the user's relatives will be notified via email and / or SMS, with a message such as "The user has received a potentially fraudulent call. Please check."

[1366] 6. How to save data after a call ends

[1367] On the device: After the call ends, the analyzed text data and the log file containing the fraud detection result are stored in a secure storage location. This storage process is performed in a secure manner so that it can be accessed by the user.

[1368] 7. A means of storing the analyzed data in secure storage

[1369] Device: Saved log files are stored in secure storage on the device and can be viewed by the user or their relatives if necessary.

[1370] Specific examples

[1371] One day, the user receives a call on their mobile phone from someone claiming to be from a public institution. As soon as the call begins, the device begins capturing the call content in real time. The local AI model converts the voice data into text and analyzes the text data. If the call content contains keywords associated with fraud, such as "unpaid bills" or "transfers," the local AI model determines that it is likely a scam. The device then immediately issues an audio warning to the user and simultaneously sends a notification to their relatives. After the call ends, the suspected fraudulent content of the call and the fraud determination result are recorded in a log file and securely stored. This series of processes significantly reduces the risk of elderly people falling victim to fraud.

[1372] Prompt Sentence Examples

[1373] "Please explain the system to be followed when an elderly person receives a fraudulent phone call."

[1374] The system can provide peace of mind to seniors and their families through real-time voice analysis and immediate response.

[1375] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1376] Step 1:

[1377] When a user answers a call, the call begins. The device detects the start of the call and begins capturing audio data using the built-in microphone. The input at this time is the audio signal during the call, and the output is audio data. Specifically, the built-in microphone of the device collects the telephone voice and converts it into digital audio data.

[1378] Step 2:

[1379] The device converts the captured voice data into text data in real time. This uses a local AI model speech recognition engine (e.g., Google Speech-to-Text or Microsoft Azure Cognitive Services). The input is voice data, and the output is text data. Specifically, the speech recognition engine analyzes voice patterns and converts them into corresponding text data.

[1380] Step 3:

[1381] The device analyzes the text data to assess the likelihood of fraud. A local AI model, the Natural Language Processing (NLP) engine, is responsible for this process. The input is the text data, and the output is a flag indicating the likelihood of fraud (a fraud alert flag). Specifically, the NLP engine filters keywords and phrases in the text to detect patterns associated with fraud.

[1382] Step 4:

[1383] When the fraud alert flag is activated, the terminal immediately issues an audio warning to the user. The input is the fraud alert flag and the output is the warning message. Specifically, the terminal plays the audio message "This call may be fraudulent. Do not continue the call."

[1384] Step 5:

[1385] When the fraud alert flag is activated, the server notifies the relative of a possible fraud. The input is a signal sent from the device to the server, and the output is a notification message to the relative. Specifically, the server sends an email or SMS to the relative's registered contacts saying, "A user has answered a suspected fraudulent call. Please check."

[1386] Step 6:

[1387] When the user ends the call, the device saves the call content and the fraud detection result in a secure log file. The input is a call end signal and text data containing the fraud detection result, and the output is a secure log file. Specifically, the device encrypts the data and saves it in secure storage.

[1388] Step 7:

[1389] The saved log file is stored in secure storage within the device. The input is a secure log file, and the output is saved in secure storage. Specifically, the device saves the log file according to a specified security protocol and allows the user or relatives to access it as needed.

[1390] (Application example 1)

[1391] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1392] There is a need to effectively reduce the risk of elderly people becoming victims of telephone fraud. However, it is difficult for elderly people to recognize the signs of fraud themselves, and they are unable to act quickly without appropriate warnings and notifications. To solve these problems, a system that processes voice data in real time and provides immediate warnings is required.

[1393] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1394] In this invention, the server includes a means for acquiring voice data in real time, a means for converting the acquired voice data into text data, a means for analyzing the possibility of fraud from the converted text data, a means for notifying relatives of the possibility of fraud if there is a possibility of fraud, and a means for issuing visual and audio warnings, thereby significantly reducing the risk of elderly people becoming victims of telephone fraud and enabling prompt action.

[1395] "Audio data" refers to recordings of voice during a call, etc., and is information that is subject to signal processing.

[1396] "Means of acquisition in real time" refers to technical means for collecting voice data immediately without delay.

[1397] "Text data" is voice data converted into character information, and is basic data for analysis.

[1398] "Means for converting" refers to the speech recognition technology used to convert voice data into text data.

[1399] "Means for analyzing" refers to algorithms and techniques used to process the captured and converted text data to assess the likelihood of fraud.

[1400] "Alert Measures" refers to the audio and visual interfaces used to notify the user of potential fraud.

[1401] "Means of notifying relatives of possible fraud" refers to notification techniques that use email, SMS, or other means to inform a user's relatives of possible fraud.

[1402] "Visual and audio warning means" refers to devices and technologies that display and transmit visual and audio warnings that are intuitively understandable to the user.

[1403] A "local terminal" is a device distributed to a user, and is an apparatus that has computational resources and storage for real-time processing.

[1404] "Local AI model" refers to an artificial intelligence model that runs within the device and incorporates learning algorithms to detect signs of fraud.

[1405] "Means for detecting the start of a call" refers to a technical means for recognizing that a telephone call has started and immediately starting to capture voice data.

[1406] This invention is a system for analyzing voice data in real time and providing warnings and notifications, designed to prevent elderly people from becoming victims of telephone fraud. The system of the present invention has the following configuration in particular.

[1407] System configuration

[1408] The system includes means for acquiring voice data in real time, means for converting the acquired voice data into text data, means for analyzing the converted text data for the possibility of fraud, means for issuing an alert when there is a possibility of fraud, means for notifying relatives of the possibility of fraud, and means for issuing visual and audio alerts, thereby enabling the user to immediately recognize the risk of fraud and the relatives to quickly respond to the crisis.

[1409] Program processing

[1410] Initial Setup

[1411] User: The user installs the "Phone Fraud Prevention App" on their device. After installation, they register the contact information of their relatives on a simple setup screen.

[1412] Server: The server receives user registration information and performs authentication and configuration.

[1413] Real-time call analysis

[1414] Terminal (smart glasses): When a call starts, the microphone built into the smart glasses captures voice data in real time.

[1415] Terminal: The acquired voice data is converted into text data using a voice recognition engine (e.g., Google API).

[1416] On-device (local AI model): Analyzes the converted text data to see if it contains fraudulent words or phrases. This uses the ScamPredictionModel.

[1417] Fraud detection and notification

[1418] Device (local AI model): Activate fraud alert flag if fraud is determined to be highly likely.

[1419] Device: If the fraud alert flag is activated, the user will be warned audibly, "This call may be fraudulent. Do not continue the call." A visual warning will also be displayed on the smart glasses display.

[1420] Server: At the same time, notify the relatives of the possible fraud via email or SMS (using the smtplib library).

[1421] Ending a call and saving a log

[1422] User: The user ends the call.

[1423] Device: After the call is completed, the call details and fraud detection results are saved in a secure log file.

[1424] On the device: The saved logs are stored in secure storage and can be viewed by the user or a relative if necessary.

[1425] Specific examples

[1426] One day, the user receives a call through their smart glasses from someone claiming to be from a government agency. As soon as the call begins, the smart glasses begin capturing the call content in real time. The local AI model converts the audio data into text and analyzes the call content. If the call content contains keywords associated with fraud, such as "unpaid bills" or "transfers," the local AI model determines that it is likely a scam. The smart glasses then immediately issue an audio and visual warning to the user and simultaneously send an emergency notification to their relatives. When the call ends, the suspected fraudulent call content and the fraud determination result are logged and securely stored.

[1427] Generative AI model prompt example

[1428] "Develop an AI model to identify potentially fraudulent phone call content. The model will take Japanese text data as input and determine whether it is likely to be fraudulent. Examples of phrases include: "unpaid money", "transfer". The output should be a Boolean value indicating the likelihood of fraud."

[1429] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1430] Step 1:

[1431] User initiates a call

[1432] Input: A user receives a call through their smart glasses.

[1433] How it works: The smart glasses' call start detection sensor recognizes the start of a call and starts collecting voice data.

[1434] Output: A call initiation signal is sent to the processing unit of the smart glasses.

[1435] Step 2:

[1436] Acquiring audio data

[1437] Input: Call initiation signal and call audio.

[1438] How it works: The smart glasses have a built-in microphone that captures audio data in real time, which is then converted into digital form and sent to a processing unit.

[1439] Output: Digital signal of audio data.

[1440] Step 3:

[1441] Converting audio data to text

[1442] Input: Digital signal of audio data.

[1443] How it works: Using a speech recognition engine (e.g. Google API) within the smart glasses, the voice data is converted into text data. During this conversion process, the voice signal is analyzed and converted into corresponding text information.

[1444] Output: Text data.

[1445] Step 4:

[1446] Text data analysis

[1447] Input: Text data.

[1448] How it works: A local AI model (ScamPredictionModel) is used to analyze text data. The analysis process detects keywords and context within the text data and assesses its likelihood of fraud.

[1449] Output: A flag indicating whether fraud is likely.

[1450] Step 5:

[1451] Fraud Alert Verification

[1452] Input: Possible fraud flag.

[1453] How it works: If the fraud alert flag is active, the system determines that fraud is likely.

[1454] Output: Alert signal.

[1455] Step 6:

[1456] Sending a warning to the user

[1457] Input: Alert signal.

[1458] How it works: The smart glasses provide users with visual and audio warnings, including a message on the display that reads "This call may be fraudulent. Do not continue the call," as well as an audio warning.

[1459] Output: A warning notice to the user.

[1460] Step 7:

[1461] Sending notifications to relatives

[1462] Input: Alert signal.

[1463] What it does: The server sends emergency notifications via email or SMS based on the contact information of the next of kin configured. It uses the smtplib library to send the notifications.

[1464] Output: Notification sent to relatives.

[1465] Step 8:

[1466] Call content and result logging

[1467] Input: call termination signal, call content, fraud detection result.

[1468] How it works: When a user ends a call, the device saves the call and the fraud detection result in a secure log file that is encrypted and stored in secure storage on the user's device.

[1469] Output: Secure log file.

[1470] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1471] This invention is a system that performs real-time analysis of voice data and provides warnings and notifications to protect elderly people from becoming victims of telephone fraud. In particular, by combining this system with an emotion engine that recognizes the user's emotions, this system enables flexible responses according to the user's emotional state.

[1472] System configuration

[1473] 1. A means of acquiring audio data in real time

[1474] Device: The user's device is equipped with the capability to capture phone conversations in real time.

[1475] 2. A means of converting the acquired voice data into text data

[1476] Device (local AI model): A speech recognition engine converts voice data into text data in real time. This conversion process is performed within the device.

[1477] 3. A method for analyzing the possibility of fraud from the converted text data

[1478] On-device (local AI model): The analytics module analyzes the converted text data and filters it to detect fraudulent words and phrases. It uses natural language processing (NLP) techniques to understand the context and assess the likelihood of fraud.

[1479] 4. A way to flag potential fraud

[1480] Terminal: If a call is deemed potentially fraudulent, the terminal will issue an audio warning to the user, such as a warning message saying, "This call may be fraudulent. Do not continue the call."

[1481] 5. How to notify relatives of potential fraud

[1482] Server: Notify the user's relatives of the potential scam via email or SMS. This notification may include a message such as "The user has received a potentially fraudulent call. Please check."

[1483] 6. Means of recognizing user emotions using an emotion engine

[1484] The device is equipped with an emotion engine that analyzes the user's emotional state based on voice data. The emotion engine analyzes the tone of voice and word choice to gauge the user's emotions (e.g., surprise, confusion, anger, etc.).

[1485] 7. A means for the emotion engine to tailor warning content based on the user's emotional state

[1486] Terminal: The emotion engine recognizes the user's emotional state and issues a warning message according to the user's emotion. For example, if the user is surprised, it will provide a gentle warning message such as "Please stay calm. This call may be a scam."

[1487] 8. A method for detailing notifications to relatives based on the user's emotional state

[1488] Device: If the user's emotional state exceeds a certain threshold, the notification to the next of kin will be detailed. For example, if the user is very distressed, the next of kin will receive a detailed message saying, "The user is very distressed. Please contact us immediately."

[1489] Program processing

[1490] Initial Setup

[1491] User: The user installs the "Phone Fraud Prevention App" on their device. After installation, they register the contact information of their relatives and the method of notification in case of an emergency on the settings screen.

[1492] Server: The server receives the user registration information, performs an authentication process, and, if successful, stores the registration information in a secure database.

[1493] Real-time call analysis and emotion recognition

[1494] User: When a call comes in to the user's phone, answer the call.

[1495] Terminal: When a call starts, the terminal starts capturing audio data in real time, which is temporarily stored in the terminal's buffer memory.

[1496] Device (local AI model): The speech recognition engine converts voice data into text data in real time.

[1497] Terminal (local AI model): The analysis module receives the converted text data and filters it to analyze possible fraud.

[1498] On-device (local AI model): A natural language processing engine understands the context and assesses the likelihood of fraud.

[1499] Terminal (emotion engine): Analyzes the user's emotional state based on voice data and recognizes emotions such as surprise, confusion, or anger.

[1500] Fraud detection and emotion-based warnings and notifications

[1501] Device (local AI model): Activate fraud alert flag if fraud is determined to be highly likely.

[1502] Terminal: When the fraud alert flag is activated, the emotion engine recognizes the user's emotional state and issues a warning message, such as "This call may be fraudulent. Do not continue the call."

[1503] On the device: At the same time, an email or SMS is sent to a relative notifying them of the possible scam. If the user's emotional state is very distressed, the notification will include a detailed message such as "The user is very distressed. Please contact us immediately."

[1504] Ending a call and saving a log

[1505] User: The user receives an alert and ends the call.

[1506] Device: After the call is completed, the call details and fraud detection results are saved in a secure log file, which is stored in secure storage on the user's device.

[1507] On the device: If necessary, the user or a relative can review the log file, which contains a portion of the call (where fraud is suspected) and details of the fraud determination.

[1508] Specific examples

[1509] One day, a user receives a call from someone claiming to be from a bank. As soon as the call begins, the device begins capturing the conversation in real time. A local AI model converts the voice data into text and analyzes the conversation. If the call contains keywords associated with fraud, such as "account number confirmation required," "unpaid bill," or "transfer," the emotion engine analyzes the user's tone of voice and word choice and recognizes that the user is confused. In this case, the device issues a warning message to the user: "Please remain calm. This call may be fraudulent. Do not continue the call." At the same time, a detailed email is sent to the user's relatives, stating, "The user is very confused. Please contact us immediately." After the user ends the call, the conversation and the fraud detection result are securely stored as a log. This process provides better protection for the elderly against telephone fraud.

[1510] The system performs real-time voice analysis as well as user emotion recognition, detecting fraud and taking appropriate action to provide peace of mind to seniors and their families.

[1511] The processing flow will be explained below.

[1512] Step 1:

[1513] User: The user installs the "Phone Fraud Prevention App" on their device. After installation, they launch the app and register on the initial setup screen. The user enters contact information for relatives and emergency notification methods to complete the setup.

[1514] Step 2:

[1515] Server: The server receives the user registration information and performs an authentication process. If authentication is successful, the user registration information is stored in a secure database.

[1516] Step 3:

[1517] User: When a call comes in to the user's phone, the user answers the call.

[1518] Step 4:

[1519] Terminal: When a call starts, the terminal starts capturing audio data in real time, which is temporarily stored in the terminal's buffer memory.

[1520] Step 5:

[1521] Device (local AI model): The speech recognition engine converts voice data into text data in real time. This conversion process is performed on the device, and the converted text data is sent to the analysis module.

[1522] Step 6:

[1523] On-device (local AI model): The analytics module receives the text data and filters it to determine whether it is potentially fraudulent, checking to see if specific fraud-related keywords or phrases are detected.

[1524] Step 7:

[1525] On-device (local AI model): Utilizing advanced natural language processing (NLP) to understand context and perform a detailed assessment of the likelihood of fraud. If fraud is deemed likely, a fraud alert flag is raised.

[1526] Step 8:

[1527] Device (Emotion Engine): Analyzes the user's emotional state using voice data. The emotion engine recognizes the user's emotional state (surprise, confusion, anger, etc.) based on the user's tone of voice and word choice.

[1528] Step 9:

[1529] Terminal: If the emotion engine determines that the user's emotion is negative, such as surprise or confusion, it generates a gentle warning message, such as "Please stay calm. This call may be fraudulent. Do not continue the call."

[1530] Step 10:

[1531] Terminal: If the fraud alert flag is activated, the terminal will audibly reproduce the generated warning message to the user.

[1532] Step 11:

[1533] Device: At the same time, promptly notify relatives, for example by sending an email or SMS with detailed information such as "User is receiving a scam call. This is particularly disturbing. Please contact them immediately," depending on the user's emotional state.

[1534] Step 12:

[1535] User: The user is warned and ends the call. Users who hear the warning are encouraged to end the call immediately.

[1536] Step 13:

[1537] Device: After the call is completed, the call details and fraud detection results are saved to a secure log file, which is stored in secure storage on the user's device.

[1538] Step 14:

[1539] Device: The user or a relative can check this log file as needed. The log file contains detailed records of suspected fraudulent calls and their evaluation results.

[1540] This provides real-time protection for seniors against phone scams, allows for flexible responses based on emotional state, and facilitates rapid response in emergencies by notifying relatives.

[1541] Example 2

[1542] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1543] To prevent elderly people from becoming victims of telephone fraud, a system that can quickly detect potential fraud and issue a warning is essential. However, conventional systems issue a uniform warning message without taking into account the user's emotional state, which can lead to users being unable to respond appropriately. Furthermore, notifications to relatives are also sent in a uniform manner, which creates an issue where the urgency of the situation is not accurately conveyed.

[1544] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for acquiring voice data in real time, means for converting the acquired voice data into text data, means for analyzing the possibility of fraud from the converted text data, means for issuing a warning when there is a possibility of fraud, means for recognizing the subscriber's emotion, means for adjusting the content of the warning depending on the recognized emotional state, and means for notifying relatives of the possibility of fraud. This enables flexible response according to the user's emotional state, effectively protecting the elderly from telephone fraud, and enabling accurate notification to relatives.

[1545] "Audio data" is data that represents an audio signal in digital form and is used to record the contents of a call.

[1546] "Real-time" refers to processing that occurs immediately without delay, meaning that voice data is acquired and processed immediately during a call.

[1547] "Text data" is data that expresses character information in digital format, and is obtained by converting voice data into a string of characters.

[1548] "Fraud likelihood" refers to assessing the likelihood that a call contains fraudulent activity and determining whether the risk is high or low.

[1549] A "warning" is a message provided by voice, text, or other means to alert the user when there is a high possibility of fraud.

[1550] "Subscribers" refer to individuals who use the system, which is primarily targeted at elderly people.

[1551] "Emotions" refer to an individual's psychological state or feelings, and specifically include states such as surprise, confusion, and anger.

[1552] "Emotional state" indicates the user's current emotional state, as analyzed by the emotion engine.

[1553] "Relatives" are family members or close relatives of the user, who are to be notified in the event of an emergency.

[1554] "Notification" is a means of conveying important information to others, and is sent to relatives in the form of email, SMS, etc.

[1555] MODE FOR CARRYING OUT THE INVENTION

[1556] System Overview

[1557] This invention is a system that performs real-time analysis of voice data and provides warnings and notifications to protect elderly people from becoming victims of telephone fraud. By combining this system with an emotion engine that recognizes the user's emotions, it enables flexible responses according to the user's emotional state.

[1558] Hardware and Software Configuration

[1559] 1. A means of acquiring audio data in real time

[1560] Device: The user's device is equipped with the capability to capture the contents of phone conversations in real time.

[1561] 2. A means of converting the acquired voice data into text data

[1562] Device (local AI model): The speech recognition engine converts voice data into text data in real time. This conversion process is performed within the device.

[1563] 3. A method for analyzing the possibility of fraud from the converted text data

[1564] On-device (local AI model): The analysis module analyzes the converted text data and filters it to detect fraudulent words and phrases. It uses natural language processing (NLP) techniques to understand the context and assess the likelihood of fraud.

[1565] 4. A way to flag potential fraud

[1566] Terminal: If a fraudulent call is detected, the terminal will issue an audio warning to the user, such as a warning message saying, "This call may be fraudulent. Do not continue the call."

[1567] 5. How to notify relatives of potential fraud

[1568] Server: The user's relatives are notified by email or SMS of a possible scam. This notification may include a message such as "The user has received a potentially scam call. Please check."

[1569] 6. Means of recognizing user emotions using an emotion engine

[1570] The device is equipped with an emotion engine that analyzes the user's emotional state based on voice data. The emotion engine analyzes the tone of voice and word choice to gauge the user's emotions (e.g., surprise, confusion, anger, etc.).

[1571] 7. A means for the emotion engine to tailor warning content based on the user's emotional state

[1572] Terminal: The emotion engine recognizes the user's emotional state and issues a warning message according to the user's emotion. For example, if the user is surprised, it will provide a gentle warning message such as "Please stay calm. This call may be a scam."

[1573] 8. A method for detailing notifications to relatives based on the user's emotional state

[1574] Device: If the user's emotional state exceeds a certain threshold, the notification to the relatives will be detailed. For example, if the user is very distressed, the relatives will receive a detailed message such as "The user is very distressed. Please contact us immediately."

[1575] Program processing explanation

[1576] Initial Setup

[1577] User: Install the "Phone Fraud Prevention App" on the device. After installation, register the contact information of relatives and the notification method in case of an emergency on the settings screen.

[1578] Server: Receives the user's registration information, performs the authentication process, and, if successful, stores the registration information in a secure database.

[1579] Real-time call analysis and emotion recognition

[1580] User: When a call comes in to the user's phone, answer the call.

[1581] Terminal: When a call starts, the terminal starts capturing voice data in real time, which is temporarily stored in the terminal's buffer memory.

[1582] Device (local AI model): The voice recognition engine converts voice data into text data in real time.

[1583] Terminal (local AI model): The analysis module receives the converted text data and filters it to analyze the possibility of fraud.

[1584] On-device (local AI model): The NLP engine understands the context and assesses the likelihood of fraud.

[1585] Terminal (emotion engine): Analyzes the user's emotional state based on voice data and recognizes emotions such as surprise, confusion, or anger.

[1586] Specific examples

[1587] Specific operation scenario

[1588] One day, the user receives a call from someone claiming to be from the bank. As soon as the call begins, the device begins capturing the conversation in real time. A local AI model converts the voice data into text and analyzes the conversation. If the call contains keywords associated with fraud, such as "account number confirmation required," "unpaid bill," or "transfer," the emotion engine analyzes the user's tone of voice and word choice and recognizes that the user is confused. In this case, the device issues a warning message to the user: "Please remain calm. This call may be fraudulent. Do not continue the call." At the same time, a detailed email is sent to the user's relatives stating, "The user is very confused. Please contact us immediately." After the user ends the call, the conversation and the fraud determination results are securely stored as logs.

[1589] Example prompts for generative AI models

[1590] "Generate natural language explanations for a system that utilizes an emotion engine to detect elderly phone scams in real time and issue appropriate warnings and notifications. The system analyzes voice data, recognizes emotions, alerts, and notifies."

[1591] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1592] Step 1: Initial Setup

[1593] User: The user installs the "Phone Fraud Prevention App" on their device. After installation is complete, they register the contact information of their relatives and the notification method in case of an emergency on the settings screen. The input data is the contact information of their relatives and the notification method, and the setting information is saved on the device as output.

[1594] Server: The server receives the registration information from the user and performs an authentication process. The input data is the user's registration information, and if the authentication is successful, the output is the registration information stored in a secure database.

[1595] Step 2: Detecting an incoming call

[1596] User: When a call comes in, the user answers the call.

[1597] Terminal: The terminal detects an incoming call and starts capturing the call content in real time when the call starts. The input data is the call start signal, and the output is the start of voice data capture. This operation is stored in buffer memory.

[1598] Step 3: Convert audio data to text

[1599] Terminal (local AI model): The speech recognition engine converts voice data during a call into text data in real time. The input data is voice data, and text data is generated as output. The specific operation is to apply a speech-to-text conversion algorithm.

[1600] Step 4: Fraud Potential Analysis

[1601] Terminal (Local AI Model): The analysis module receives the generated text data and filters out fraudulent words and phrases. The input data is text data, and the output is a determination of whether or not it is likely to be fraudulent. This filtering is performed using keyword search and NLP techniques.

[1602] Step 5: Emotion Recognition

[1603] Terminal (Emotion Engine): Analyzes the user's emotional state based on voice data. The input data is voice and text data, and the output recognizes the user's emotional state (surprise, confusion, anger, etc.). Specific operations include an emotion recognition algorithm that analyzes tone and word choice.

[1604] Step 6: Fraud detection

[1605] Terminal (local AI model): If it determines that there is a high probability of fraud, it activates a fraud alert flag. The input data are the filtering results and emotional state, and the output is a fraud alert flag.

[1606] Step 7: Sending an alert

[1607] Terminal: When the fraud alert flag is activated, a warning message is issued according to the user's emotional state as recognized by the emotion engine. The input data is the fraud alert flag and the emotional state, and the warning message is generated as the output. Specific operations include, for example, outputting a message such as "This call may be fraudulent. Please do not continue the call."

[1608] Step 8: Notify your next of kin

[1609] Terminal: If a fraud is deemed likely, a notification is sent to relatives via email or SMS. The input data is the fraud alert flag and the emotional state, and the output is a notification message. Specific actions include sending a detailed notification such as "The user is very confused. Please contact us immediately."

[1610] Step 9: End the call and save the log

[1611] User: Receives warning and ends call.

[1612] Terminal: After the call ends, the call content and fraud determination results are saved in a secure log file. The input data is the call content and fraud determination results, and log data is generated as output. This log data is stored in secure storage within the terminal. Specifically, the log file records part of the call content (the part suspected of being fraudulent) and details of the fraud determination.

[1613] In this way, the system can effectively protect the elderly from telephone fraud by performing real-time analysis combined with user emotion recognition.

[1614] (Application example 2)

[1615] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1616] This invention relates to a system for preventing elderly people from becoming victims of telephone fraud. In particular, it aims to enhance user safety by not only analyzing the possibility of fraud but also recognizing the user's emotions and responding flexibly according to their state. Current systems only detect fraud and do not take into account the user's emotional state, which often leaves elderly people confused and frightened. For this reason, there is a need for the development of a system that also incorporates emotion recognition.

[1617] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for acquiring voice data in real time, means for converting the acquired voice data into text data, means for analyzing the possibility of fraud from the converted text data, means for issuing a warning when there is a possibility of fraud, means for notifying relatives of the possibility of fraud, means for recognizing the user's emotions, means for adjusting the content of the warning depending on the user's emotional state, and means for detailing the content of the notification depending on the emotional state. This reduces the possibility that elderly people will become victims of telephone fraud and enables appropriate responses depending on the user's emotional state.

[1618] "Voice data" refers to data that is recorded or acquired in digital form, such as voice from a conversation.

[1619] "Real-time" means that the processing occurs immediately, without delay.

[1620] "Means" refers to a method or apparatus for achieving a particular function.

[1621] "Text data" is digital data that has been converted from voice data into character information.

[1622] "Fraud likelihood analysis" refers to assessing the likelihood of fraudulent activity based on voice and text data.

[1623] "Issuing a warning" refers to the act of issuing a warning or alert to the user.

[1624] "Relatives" refers to an individual's family or blood relatives, including parents, siblings, and children.

[1625] "Emotion recognition" means analyzing and determining a user's psychological state and emotions (for example, surprise, confusion, anger, etc.) from voice or text.

[1626] "Adjusting the warning content according to the emotional state" means generating a warning message with appropriate expressions and tone, taking into account the emotional state of the user.

[1627] "Detailing the notification content" means making the specific information and content of the notification to relatives more detailed and precise depending on the user's emotional state and the possibility of fraud.

[1628] "Local terminal" refers to a local computer or device that can be directly operated by a user.

[1629] A "local AI model" is an artificial intelligence model that is placed within a specific enclosure or device and operates independently of the internet.

[1630] This invention is a system to protect elderly people from becoming victims of telephone fraud. In particular, it not only analyzes the possibility of fraud but also recognizes the user's emotions and responds flexibly according to their emotional state.

[1631] System configuration

[1632] 1. Real-time acquisition of audio data

[1633] Terminal: Equipped with a function to capture voice data during a call in real time. Specifically, it uses the smartphone's microphone.

[1634] 2. Converting audio data to text data

[1635] On the device: Use the Google Speech-to-Text API to convert captured audio data into text data.

[1636] 3. Analyze for potential fraud

[1637] Terminal: Use spaCy as a natural language processing engine to analyze possible fraud from the converted text data.

[1638] 4. Emotion recognition

[1639] Device: Uses IBM Watson Tone Analyzer to analyze the user's emotional state based on voice and text data.

[1640] 5. Alerts and Notifications

[1641] Terminal: If there is a high possibility of fraud, the system will warn the user and send a notification to their relatives. The warning message will be generated using the results of the emotion engine.

[1642] Server: To notify relatives, we use Twilio to send SMS and SMTP to send email.

[1643] 6. Log storage

[1644] On the device: Call logs and analytics are stored in a SQLite database for secure use.

[1645] Specific examples

[1646] One day, the user receives a call on their smartphone from someone claiming to be from the bank. As soon as the call begins, the device begins capturing the call in real time, and the audio data is converted into text using the Google Speech-to-Text API. If the converted text contains keywords that suggest fraud, such as "account number confirmation required," "unpaid balance," or "transfer," the device analyzes the text data with spaCy and determines that there is a high possibility of fraud. At the same time, the device uses IBM Watson Tone Analyzer to analyze the user's emotional state and recognizes that the user is feeling confused or surprised.

[1647] Based on the results, the device will display a soft warning message to the user, such as "Please stay calm. This call may be fraudulent. Do not continue the call." Additionally, Twilio will be used to send a detailed email notification to the next of kin, such as "The user is very distressed. Please contact us immediately."

[1648] This series of processes is extremely effective in strengthening user security and preventing fraud before it occurs. In addition, the information saved as a log can be used for future verification and as evidence.

[1649] Example prompts for generative AI models

[1650] Write a program that converts voice data into text data, analyzes the text data, and detects possible fraud. This program uses IBM Watson Tone Analyzer to recognize the user's emotions, and if fraud is detected, sends an SMS to relatives using the Twilio API and an email using SMTP. Also, save the call content and analysis results in a SQLite database.

[1651] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1652] Step 1:

[1653] The device acquires voice data in real time. This involves using the smartphone's microphone to capture voice data from the beginning to the end of a call. The input is the voice during the call, and the output is the captured voice data.

[1654] Step 2:

[1655] The device converts the acquired voice data into text data. The Google Speech-to-Text API is used to convert the voice data into text format. The input is voice data and the output is text data. This conversion allows the spoken content to be handled as text information.

[1656] Step 3:

[1657] The device analyzes the converted text data to detect potential fraud. It uses a natural language processing engine (e.g., spaCy) to assess the likelihood of fraud based on specific keywords and phrases. The input is the text data, and the output is a fraud assessment result (whether fraud is likely or unlikely). This analysis identifies indicators of fraudulent activity.

[1658] Step 4:

[1659] An emotion recognition engine is used to analyze the user's emotional state. IBM Watson Tone Analyzer is used to determine the user's emotion (for example, surprise, confusion, anger, etc.) based on text data and voice tone. The input is voice data and text data, and the output is the result of the emotional state determination. This allows the user's psychological state to be understood.

[1660] Step 5:

[1661] If the device is determined to be highly likely to be a fraudster, a warning is issued to the user. Based on the emotion recognition results, a warning message is generated in an appropriate tone. For example, it could say, "This call may be fraudulent. Please do not continue the call." The input is the fraud detection result and the emotional state detection result, and the output is the warning message. This allows the user to be immediately alerted.

[1662] Step 6:

[1663] The device notifies relatives of possible fraud. It uses the Twilio API to send an SMS and an email via SMTP. This notification contains detailed information based on the user's emotional state. For example, it could say, "The user is very confused. Please contact us immediately." The input is the fraud detection result and the emotional state detection result, and the output is the SMS and email sent. This allows relatives to respond quickly.

[1664] Step 7:

[1665] The device saves the call content and analysis results in a secure log file. An SQLite database is used to record the call content, fraud detection results, and emotional state detection results. The input is the text data of the call content, fraud detection results, and emotional state detection results, and the output is the saved log file. This makes it possible to check the call content and fraud detection results later.

[1666] Step 8:

[1667] The user or a relative can check the log file as needed. The log file contains a portion of the call content (parts suspected of fraud) and the analysis results. The input is the request, and the output is the display of the log file. This allows the user to check the details of past call content and the fraud detection status.

[1668] Prompt Sentence Examples

[1669] Write a program that converts voice data into text data, analyzes the text data, and detects possible fraud. This program uses IBM Watson Tone Analyzer to recognize the user's emotions, and if fraud is detected, sends an SMS to relatives using the Twilio API and an email using SMTP. Also, save the call content and analysis results in a SQLite database.

[1670] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1671] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1672] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1673] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1674] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1675] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1676] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1677] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1678] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1679] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1680] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1681] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1682] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1683] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1684] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1685] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1686] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1687] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1688] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1689] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1690] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1691] The following is further disclosed regarding the above embodiment.

[1692] (Claim 1)

[1693] a means for acquiring audio data in real time;

[1694] means for converting the acquired voice data into text data;

[1695] means for analyzing the converted text data for possible fraud;

[1696] a means of alerting you to potential fraud;

[1697] A system that includes a means to notify relatives of potential fraud.

[1698] (Claim 2)

[1699] means for processing the captured audio data on the local terminal;

[1700] 10. The system of claim 1, including a local AI model that performs a likelihood of fraud analysis.

[1701] (Claim 3)

[1702] 2. The system according to claim 1, wherein the means for acquiring voice data in real time is means for detecting the start of a call.

[1703] "Example 1"

[1704] (Claim 1)

[1705] a means for acquiring audio data in real time;

[1706] means for converting the acquired voice data into text data;

[1707] means for analyzing the converted text data for possible fraud;

[1708] a means of alerting you to potential fraud;

[1709] a means of notifying relatives of possible fraud;

[1710] a means for storing data after the call has ended;

[1711] The system includes means for storing the analyzed data in secure storage.

[1712] (Claim 2)

[1713] means for processing the captured audio data on the local terminal;

[1714] 10. The system of claim 1, including a local AI model that performs a likelihood of fraud analysis.

[1715] (Claim 3)

[1716] 2. The system according to claim 1, wherein the means for acquiring voice data in real time is means for detecting the start of a call.

[1717] "Application Example 1"

[1718] (Claim 1)

[1719] a means for acquiring audio data in real time;

[1720] means for converting the acquired voice data into text data;

[1721] means for analyzing the converted text data for possible fraud;

[1722] a means of alerting you to potential fraud;

[1723] a means of notifying relatives of possible fraud;

[1724] A system that includes visual and audio warning means.

[1725] (Claim 2)

[1726] means for processing the captured audio data on the local terminal;

[1727] 10. The system of claim 1, including a local AI model that performs a likelihood of fraud analysis.

[1728] (Claim 3)

[1729] 2. The system according to claim 1, wherein the means for acquiring voice data in real time is means for detecting the start of a call.

[1730] "Example 2: Combining Emotion Engines"

[1731] (Claim 1)

[1732] a means for acquiring audio data in real time;

[1733] means for converting the acquired voice data into text data;

[1734] means for analyzing the converted text data for possible fraud;

[1735] a means of alerting you to potential fraud;

[1736] means for recognizing subscriber emotions;

[1737] means for tailoring the alert content in response to the perceived emotional state;

[1738] A system that includes a means to notify relatives of potential fraud.

[1739] (Claim 2)

[1740] means for processing the captured audio data on the local terminal;

[1741] 10. The system of claim 1, including a local AI model that performs a likelihood of fraud analysis.

[1742] (Claim 3)

[1743] 2. The system according to claim 1, wherein the means for acquiring voice data in real time is means for detecting the start of a call.

[1744] "Application example 2 when combining emotion engines"

[1745] (Claim 1)

[1746] a means for acquiring audio data in real time;

[1747] means for converting the acquired voice data into text data;

[1748] means for analyzing the converted text data for possible fraud;

[1749] a means of alerting you to potential fraud;

[1750] a means of notifying relatives of possible fraud;

[1751] means for recognizing a user's emotion;

[1752] means for adjusting the content of the warning according to the emotional state of the user;

[1753] The system includes a means for refining notification content depending on emotional state.

[1754] (Claim 2)

[1755] means for processing the captured audio data on the local terminal;

[1756] A local AI model that analyzes potential fraud,

[1757] 10. The system of claim 1, including a local AI model for user emotion recognition.

[1758] (Claim 3)

[1759] 2. The system according to claim 1, wherein the means for acquiring voice data in real time is means for detecting the start of a call. [Explanation of symbols]

[1760] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for acquiring audio data in real time; means for converting the acquired voice data into text data; means for analyzing the converted text data for possible fraud; a means of alerting you to potential fraud; A system that includes a means to notify relatives of potential fraud.

2. means for processing the captured audio data on the local terminal; 10. The system of claim 1, further comprising a local AI model for performing a likelihood of fraud analysis.

3. 2. The system according to claim 1, wherein the means for acquiring voice data in real time is means for detecting the start of a call.

Images