System

Abstract

A system is provided.SOLUTION: A system comprising: means for recording audio; means for converting the recorded audio to text; means for parsing the generated text to detect certain keywords; means for sending a notification based on the detected keywords; and means for storing the detection results and audio data.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Describe the "problem that the invention aims to solve" and the "means for solving the problem."

[0005] ---

[0006] Telephone scams targeting the elderly are becoming more sophisticated every year, and the number of victims continues to grow. Elderly people are particularly susceptible to fraud, as their comprehension and judgment are often impaired. Furthermore, if an elderly person lives alone, it takes time for them to notice something unusual, increasing the risk of the damage spreading. To solve these problems, a system is needed that can detect fraudulent phone calls in real time and prevent fraud damage before it occurs. It is also important to collect and analyze this data, identify fraudulent methods early, and share it with law enforcement agencies.

[0007] --- [Means for solving the problem]

[0008] The present invention relates to a system for detecting telephone fraud in real time and notifying elderly people and their families. The above-mentioned problems are solved by using the following means.

[0009] a means for recording audio;

[0010] a means for converting recorded speech into text;

[0011] means for analyzing the generated text to detect specific keywords;

[0012] means for sending notifications based on the detected keywords;

[0013] means for storing the detection results and the audio data;

[0014] This system includes the above (Claim 1). This makes it possible to detect fraudulent phone calls in real time and prevent fraud damage before it occurs.

[0015] The system further includes a means for sending the notification to the user (claim 2). This allows family members living far away to respond quickly and prevents fraud against the elderly.

[0016] In addition, the system includes a means for sharing the analysis results and audio data with law enforcement agencies (Claim 3), which will enable analysis of trends and patterns of fraudulent activity based on the collected data and help crack down on future frauds.

[0017] ---

[0018] That's all.

[0019] ---

[0020] A "voice recording device" is a device or system that stores the contents of a telephone conversation as digital data.

[0021] ---

[0022] A "recorded speech-to-text converter" is a process or device that converts recorded speech data into natural language text using speech recognition technology.

[0023] ---

[0024] "Means for detecting specific keywords" refers to software or a system that has the ability to automatically extract and analyze pre-defined fraud-related keywords and phrases from the generated text data.

[0025] ---

[0026] "Means for sending notifications" refers to communication methods for sending warnings or information to users (seniors and their families) based on the detected potential fraud. Specifically, these include SMS, email, and dedicated apps.

[0027] ---

[0028] "Means for storing detection results and audio data" refers to a system or device for safely storing the analyzed results and recorded audio data in a database or storage.

[0029] ---

[0030] "User" refers to a person who uses the system and their family, and in particular, to an entity that receives notifications from the system for the purpose of preventing fraud.

[0031] ---

[0032] "Law enforcement agencies" are administrative or judicial agencies that enforce the law and maintain public order. Specifically, they include the police and prosecutors.

[0033] ---

[0034] That's all. [Brief explanation of the drawings]

[0035] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0036] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0037] First, the terms used in the following description will be explained.

[0038] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0039] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0040] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0041] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0042] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0043] [First embodiment]

[0044] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0045] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0046] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0047] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0048] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0049] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0050] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0051] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0052] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0053] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0054] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0055] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0056] ---

[0057] The present invention relates to a system for detecting telephone fraud in real time and notifying elderly people and their families. A specific embodiment of the system and program processing will be described below.

[0058] This system detects the contents of phone calls in real time, analyzes specific keywords, and notifies the elderly and their families. The system consists of the following main components:

[0059] 1. Terminal: A telephone terminal used by the elderly. This terminal records calls and generates audio data.

[0060] 2. Server: Receives voice data, performs speech recognition, generates text data, and analyzes and detects keywords and phrases that may be fraudulent.

[0061] 3. Users: Elderly people and their families. They receive notifications from the server and take appropriate action in case of possible fraud.

[0062] Program processing

[0063] 1. Acquiring audio data

[0064] When the device detects an incoming call, it automatically starts recording the call, and when the call ends, it temporarily saves the recorded audio data.

[0065] 2. Sending audio data

[0066] The device compresses the recorded audio data and sends it to the server using a secure protocol.

[0067] 3. Converting voice data to text

[0068] The server passes the received voice data to a voice recognition engine, converts the voice into text, and obtains the text data generated by the voice recognition engine.

[0069] 4. Natural Language Processing for Fraud Detection

[0070] The server analyzes the text data, detecting specific keywords and phrases (e.g., "transfer" or "prepaid card"), and uses natural language processing techniques to evaluate the content for potential fraud.

[0071] 5. Notification Processing

[0072] Based on the analysis results, the server sends a notification to the elderly person and their family, including the specific wording of the suspected fraudulent call and a summary of the call.

[0073] Users can receive notifications, review the details, and take immediate action, for example, allowing family members to contact the elderly and prevent fraud before it occurs.

[0074] 6. Data Storage and Sharing

[0075] The server stores the detection results and audio data in a database, allowing for detailed analysis and review later if necessary.

[0076] Where necessary, we may share collected data with law enforcement agencies to help them investigate fraud trends and patterns.

[0077] Specific examples

[0078] Detection of bank transfer fraud

[0079] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "This is my son. I urgently need money, so please transfer it to the specified account right now."

[0080] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[0081] 3. The server passes the voice data to a voice recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer it to the specified account immediately."

[0082] 4. The server analyzes the text data and detects the phrase "Transfer the money to the specified account." It determines that the message is likely to be fraudulent.

[0083] 5. The server immediately sends a notification to the user (the elderly person's family member) saying, "You have received a potentially fraudulent call. The message is, 'We urgently need money, so please transfer it to the specified account immediately.' Please be careful."

[0084] 6. The user will receive a notification and contact the elderly person to warn them and instruct them not to make the transfer.

[0085] Prepaid card fraud detection

[0086] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "Please buy a prepaid card at a convenience store and give me the number."

[0087] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[0088] 3. The server passes the voice data to a voice recognition engine, which generates text data such as, "Please purchase a prepaid card at a convenience store and tell me the number."

[0089] 4. The server analyzes the text data and detects the phrase "purchase a prepaid card." It determines that the data is likely fraudulent.

[0090] 5. The server immediately sends a notification to the user (the elderly person's family member) saying, "You have received a potentially fraudulent call. The caller asks you to purchase a prepaid card at a convenience store and provide the number. Please be careful."

[0091] 6. The user will receive a notification and contact the elderly person to warn them and provide guidance on how to prevent fraud.

[0092] Through real-time analysis and notification of voice data, this system is an effective means of protecting the safety of the elderly and is useful in combating widespread fraud.

[0093] ---

[0094] That's all.

[0095] The processing flow will be explained below.

[0096] ---

[0097] Step 1:

[0098] When the device detects an incoming call, it automatically starts recording the call, so that the conversation between the elderly person and the other party is recorded as digital audio data from the moment the call begins.

[0099] ---

[0100] Step 2:

[0101] After the call ends, the device compresses the recorded audio data to improve data transfer efficiency and save storage space. The compressed audio data is then temporarily stored on the device.

[0102] ---

[0103] Step 3:

[0104] The device uses a secure protocol to send compressed audio data to the server, using encrypted communications over the internet to ensure data privacy and security.

[0105] ---

[0106] Step 4:

[0107] The server passes the received voice data to a voice recognition engine, which converts the voice into text. The voice recognition engine then analyzes the voice data and outputs it as "character string data."

[0108] ---

[0109] Step 5:

[0110] The server then inputs the generated text data into a natural language processing (NLP) module, which analyzes the text and detects specific keywords and phrases, such as whether it contains terms associated with fraud, such as "transfer" or "prepaid card."

[0111] ---

[0112] Step 6:

[0113] The server evaluates the analysis results from the NLP module and, if it determines that there is a high possibility of fraud, begins preparing a notification. At this stage, the notification content is generated and a message is created to be sent to the elderly person and their family.

[0114] ---

[0115] Step 7:

[0116] The server sends a notification message, which can be sent to the user via SMS, email, or a dedicated app, allowing the user to quickly become aware of potential fraud and take action.

[0117] ---

[0118] Step 8:

[0119] The user receives a notification, which includes the suspected fraudulent text and a summary of the call, and can then contact the senior to provide guidance or advice.

[0120] ---

[0121] Step 9:

[0122] The server stores the analysis results and audio data in a database, which can be used for later review and analysis.

[0123] ---

[0124] Step 10:

[0125] If necessary, the server will initiate the process of sharing collected data with law enforcement agencies to investigate fraud trends and patterns and assist in legal action.

[0126] ---

[0127] That's all.

[0128] Example 1

[0129] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0130] In recent years, telephone fraud has been on the rise, with elderly people being particularly vulnerable. To prevent fraud, it is necessary to detect suspicious calls in real time and promptly notify the elderly person and their family. However, conventional systems are unable to adequately perform this real-time detection and notification, which often results in damage occurring. For this reason, there is a need for a system that can detect and notify fraud more effectively and quickly.

[0131] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0132] In this invention, the server includes a means for compressing voice data and transmitting it using a secure protocol, a means for converting recorded voice into text, a natural language processing means for analyzing the generated text and detecting specific keywords, a means for sending a notification based on the detected keywords, a means for sending the notification to the user, a means for saving the analysis results and the voice data, and a means for sharing the analysis results and the voice data with law enforcement agencies as needed. This makes it possible to detect suspected fraudulent calls in real time and quickly notify the elderly person and their family.

[0133] "Audio recording means" refers to a device or program that automatically records telephone conversations and saves the audio data.

[0134] "Means for compressing recorded audio data and transmitting it using a secure protocol" refers to a device or program that compresses recorded audio files using a compression algorithm and transmits them to a server using a secure communication protocol such as HTTPS or SSL.

[0135] A "means for converting recorded speech to text" is a system or program that uses a speech recognition engine to analyze speech data and convert it into a human-understandable text format.

[0136] "Natural language processing means for analyzing generated text and detecting specific keywords" means a system or program that uses natural language processing techniques to analyze text data and detect specific keywords or phrases that may be suspected of being fraudulent.

[0137] "Means for sending notifications based on detected keywords" means a system or program for sending warnings or notifications to users when detected keywords or phrases indicate suspected fraud.

[0138] "Means for sending notifications to users" means a system or program for sending notifications to the elderly person and their family members via SMS, email, or other communication means.

[0139] "Means for storing analysis results and audio data" refers to a system or program for storing analysis results and audio data in a database or file system, and keeping them in a state where they can be accessed and analyzed later.

[0140] "Means for sharing analytics results and audio data with law enforcement as needed" means a system or program that shares stored analytics results and audio data with law enforcement to assist in fraud investigation and prevention.

[0141] "Communication terminals used by the elderly" refer to telephones, smartphones, or other communication devices capable of making calls that are primarily used by the elderly in their daily lives.

[0142] A "server with a speech recognition engine" is a server that has speech recognition capabilities to convert received speech data into text, and is a device that has the computing resources to execute programs and algorithms to analyze speech.

[0143] The present invention relates to a system for detecting telephone fraud in real time and notifying elderly people and their families of the fraud. Specific embodiments of the system will be described below.

[0144] This system consists of the following main hardware and software:

[0145] Hardware and Software Configuration

[0146] 1. Device:

[0147] Telephones and smartphones are used as communication devices by the elderly. These devices record calls and generate audio data.

[0148] The software used is a module for recording (for example, a recording class developed in Java or Swift).

[0149] 2. Server:

[0150] A server that receives voice data, performs speech recognition, and generates text data. The server is equipped with a speech recognition engine (e.g., Google Speech-to-Text API) and a natural language processing engine (e.g., SpaCy or NLTK).

[0151] To run the program, web frameworks such as Python, Flask, and Django are used.

[0152] 3. User:

[0153] The elderly person and their family members receive notifications from the server and take appropriate action if there is a possibility of fraud.

[0154] Data processing and calculation

[0155] Acquiring audio data

[0156] When the device detects an incoming call, it automatically starts recording the call. For example, you can use the smartphone's built-in app or a dedicated recording app. When the call ends, the recorded audio data is temporarily stored on the device.

[0157] Sending audio data

[0158] The device compresses the recorded audio data and sends it to the server using a secure protocol (e.g., HTTPS or SSL) using an algorithm such as GZIP.

[0159] Converting audio data to text

[0160] The server passes the received voice data to a speech recognition engine (e.g., Google Speech-to-Text API) and converts the voice into text, which is then stored in a database.

[0161] Natural Language Processing for Fraud Detection

[0162] The server analyzes the text data using a natural language processing engine (e.g., SpaCy or NLTK) to detect specific keywords and phrases (e.g., "transfer," "prepaid card," etc.).

[0163] Notification Processing

[0164] Based on the analysis results, the server sends a notification to the elderly person and their family, including specific wording of the suspected fraud and a summary of the call. For example, notifications are sent via SMS or email using Twilio or SendGrid.

[0165] Data storage and sharing

[0166] The server stores the detection results and audio data in a database using MySQL or PostgreSQL. The stored data can be analyzed and reviewed in detail at a later date, and can be shared with law enforcement agencies if necessary.

[0167] Specific examples

[0168] Detection of bank transfer fraud

[0169] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "This is my son. I urgently need money, so please transfer it to the specified account right now."

[0170] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[0171] 3. The server passes the voice data to a voice recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer it to the specified account immediately."

[0172] 4. The server analyzes the text data and detects the phrase "Transfer the money to the specified account." It determines that the message is likely to be fraudulent.

[0173] 5. The server immediately sends a notification to the user (the elderly person's family member) saying, "You have received a potentially fraudulent call. The message is, 'We urgently need money, so please transfer it to the specified account immediately.' Please be careful."

[0174] 6. The user receives a notification and contacts the elderly person to warn them and prevent any damage from occurring.

[0175] Prepaid card fraud detection

[0176] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "Please buy a prepaid card at a convenience store and give me the number."

[0177] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[0178] 3. The server passes the voice data to a voice recognition engine, which generates text data such as, "Please purchase a prepaid card at a convenience store and tell me the number."

[0179] 4. The server analyzes the text data and detects the phrase "purchase a prepaid card." It determines that the data is likely fraudulent.

[0180] 5. The server immediately sends a notification to the user (the elderly person's family member) saying, "You have received a potentially fraudulent call. The caller asks you to purchase a prepaid card at a convenience store and provide the number. Please be careful."

[0181] 6. The user receives a notification and contacts the elderly person to warn them and prevent any damage from occurring.

[0182] Prompt Sentence Examples

[0183] In the case of detecting bank transfer fraud

[0184] "Please tell me how to build a system that notifies the elderly person and their family when they receive a call from someone pretending to be their son requesting that they transfer money to a specified account."

[0185] Prepaid card fraud detection

[0186] "Please tell me how to build a system that notifies elderly people and their families when they purchase a prepaid card at a convenience store and are asked to provide the card number."

[0187] The system provides real-time voice analysis and rapid notification to protect against fraud targeting seniors, making it effective in combating a wide range of frauds.

[0188] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0189] Step 1: Acquire audio data

[0190] When the device detects an incoming call, it automatically starts recording the call, recording the contents of the call in real time.

[0191] Input: Call audio

[0192] Data processing: Save call audio as a recording file

[0193] Output: Recording file

[0194] How it works: The device's calling app triggers an incoming call event, calls a recording module (for example, a recording class written in Java or Swift) to start recording, and then saves the recording file in the device's experience data storage after the call ends.

[0195] Step 2: Sending audio data

[0196] The device compresses the recorded audio data and sends it to the server using a secure protocol.

[0197] Input: Recording file

[0198] Data processing: Compress the audio data using GZIP or other methods and package it for HTTPS requests

[0199] Output: Compressed audio data is sent to the server

[0200] What it does: It compresses the recorded audio data using the GZIP compression algorithm and sends it to the server's API endpoint using the HTTPS protocol, for example, using Python's requests library or Java's HttpURLConnection class.

[0201] Step 3: Convert audio data to text

[0202] The server passes the received voice data to a voice recognition engine, which converts the voice into text.

[0203] Input: Compressed audio data

[0204] Data processing: Converting speech to text using a speech recognition engine (such as Google Speech-to-Text API)

[0205] Output: Text data

[0206] What happens: The server receives the audio data, decompresses it, and passes it to the speech recognition engine. The speech recognition engine analyzes the audio file and generates text data, which is then stored in a database.

[0207] Step 4: Natural Language Processing for Fraud Detection

[0208] The server analyzes the text data using a natural language processing engine to detect specific keywords and phrases.

[0209] Input: Text data

[0210] Data processing: Use natural language processing engines (such as SpaCy or NLTK) to extract and analyze specific keywords and phrases within text data.

[0211] Output: Analysis results showing suspected fraudulent phrases and keywords

[0212] How it works: The server passes the text data to a natural language processing engine, which analyzes it for potentially fraudulent keywords and phrases. Based on the analysis, any data that is suspected to be fraudulent is flagged and stored in a database.

[0213] Step 5: Notification Processing

[0214] Based on the analysis results, the server sends a notification to the elderly person and their family.

[0215] Input: Analysis results

[0216] Data processing: Generate notification content and send it using a notification service such as Twilio or SendGrid

[0217] Output: Email and SMS notifications sent to the elderly person and their family

[0218] Specific operation: The server composes the analysis results in JSON format and sends emails or SMS to the elderly person and their family members through existing notification services such as Twilio and SendGrid.

[0219] Step 6: Data storage and sharing

[0220] The server stores the detection results and audio data in a database, which may be shared with law enforcement agencies if necessary.

[0221] Input: Analysis results and audio data

[0222] Data processing: Analysis results and audio data are written to a database and shared with law enforcement agencies as needed

[0223] Output: Analysis results and audio data stored in a database

[0224] What it does: The server uses an ORM (Object-Relational Mapping) tool such as SQLAlchemy to write the analysis results and audio data to a database (such as MySQL or PostgreSQL), and provides access and data sharing to law enforcement agencies as needed.

[0225] These steps will allow the entire system to work together to protect seniors from phone scams.

[0226] (Application example 1)

[0227] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0228] In recent years, frauds targeting the elderly have been increasing. However, many elderly people do not have sufficient means to avoid fraud, and the damage caused by these frauds is becoming more serious. To solve this problem, a system is needed that can detect telephone frauds in real time and quickly notify the elderly and their families.

[0229] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0230] In this invention, the server includes a means for transmitting recorded voice data to a remote server using a security protocol, a means for converting the voice data into text using a voice recognition engine, a means for analyzing the text data using natural language processing technology, and a means for sending a push notification to a smartphone, thereby enabling real-time detection of suspected fraud and prompt notification to the elderly person and their family.

[0231] "Audio recording means" means a device or function that records the contents of a telephone conversation as digital audio data.

[0232] "Means for converting recorded voice into text" refers to a technology for converting voice data into text in a natural language, i.e., a device or function that generates text data from voice using a voice recognition engine.

[0233] "Means for analyzing the generated text and detecting specific keywords" refers to a device or function that analyzes the generated text data using algorithms or natural language processing techniques to identify keywords or phrases that may be fraudulent.

[0234] A "means for sending notifications based on detected keywords" is a device or function that, when a keyword that indicates suspected fraud is detected, notifies the user or their family of the potential fraud via a notification mechanism.

[0235] "Means for storing detection results and audio data" refers to a device or function that stores the analyzed text data and original audio data in a database or storage system so that they can be accessed later.

[0236] "Means for transmitting recorded audio data to a remote server using a secure protocol" refers to a device or function that compresses the recorded audio data and transfers it to a remote server using a secure communication protocol (e.g., HTTPS).

[0237] "Means for converting voice data into text using a voice recognition engine" refers to a device or function that mechanically converts voice data into text using voice recognition software or a cloud service.

[0238] A "means for analyzing text data using natural language processing techniques" is a device or function that uses natural language processing algorithms or models to evaluate text data and identify keywords or phrases that may contain indicators of fraud.

[0239] A "means for sending push notifications to a smartphone" is a device or system with push notification functionality for quickly notifying users and their families when potential fraud is detected.

[0240] The present invention relates to a system for detecting telephone fraud in real time and notifying elderly people and their families promptly. The following describes a specific embodiment and the hardware and software used.

[0241] System configuration

[0242] The system consists of the following main components:

[0243] 1. Device:

[0244] This is a telephone terminal used by elderly people. It records calls and generates audio data.

[0245] This includes communication devices such as smartphones with call recording functionality installed.

[0246] 2. Server:

[0247] Voice data is received, voice recognition is performed, and text data is generated.

[0248] Specific keywords are analyzed using natural language processing technology.

[0249] Google Cloud Speech-to-Text and Amazon Transcribe are used as speech recognition engines.

[0250] Natural language processing technologies used include Google Cloud Natural Language API and Amazon Comprehend.

[0251] 3. User:

[0252] The elderly themselves and their families.

[0253] Receive notifications from the server and take appropriate action in case of potential fraud.

[0254] About program processing

[0255] The processing flow by the system is shown below.

[0256] 1. Acquire audio data:

[0257] When the device detects an incoming call, it automatically records the call. When the call ends, it stops recording and temporarily saves the audio data.

[0258] 2. Sending audio data:

[0259] The device compresses the recorded audio data and sends it to the server using a secure communication protocol (e.g., HTTPS).

[0260] 3. Audio to text conversion:

[0261] The server passes the received voice data to a speech recognition engine, converts the voice into text data, and generates highly accurate text data from the voice data using Google Cloud Speech-to-Text or Amazon Transcribe.

[0262] 4. Natural Language Processing for Fraud Detection:

[0263] The server analyzes the text data using natural language processing technology (Google Cloud Natural Language API and Amazon Comprehend) to detect keywords and phrases that may be fraudulent.

[0264] 5. Notification Processing:

[0265] Based on the analysis results, the server sends a push notification to the smartphone, and also notifies the user and their family members via email and SMS.

[0266] Use Firebase Cloud Messaging or Amazon SNS for push notification.

[0267] 6. Data Storage and Sharing:

[0268] The server stores the analysis results and audio data in a cloud database (Firebase Firestore or Amazon DynamoDB), which allows the information to be shared with law enforcement agencies if necessary.

[0269] Specific examples

[0270] Example of detecting bank transfer fraud

[0271] The device detects an incoming call and records the conversation. The scammer says, "This is my son. I urgently need money, so please transfer the money to my specified account now." When the call ends, the recorded audio data is compressed and sent to the server. The server passes the audio data to a speech recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer the money to my specified account now." The server analyzes the text data and detects the phrase "transfer the money to my specified account." It immediately sends a notification to the user saying, "You have received a potentially fraudulent call. The message is, 'I urgently need money, so please transfer the money to my specified account now.' Please be careful."

[0272] Prepaid card fraud detection example

[0273] The device detects an incoming call and records the conversation. The scammer says, "Please buy a prepaid card at a convenience store and give me the number." When the call ends, the recorded audio data is compressed and sent to the server. The server passes the audio data to a speech recognition engine, which generates text data saying, "Please buy a prepaid card at a convenience store and give me the number." The server analyzes the text data and detects the phrase "please buy a prepaid card." It immediately sends a notification to the user saying, "You have received a potentially fraudulent call. The message is, 'Please buy a prepaid card at a convenience store and give me the number.' Please be careful."

[0274] In this way, this system can protect the elderly from fraud through real-time analysis and notification of voice data.

[0275] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0276] Step 1:

[0277] The device detects an incoming call and automatically starts recording the call. When the call ends, the recording stops and the audio data is temporarily saved.

[0278] Specifically, the call recording function installed on the smartphone is used to save the audio data during the call in a digital format, such as a .wav or .3gp file.

[0279] Step 2:

[0280] The device compresses the recorded audio data and sends it to the server using a secure communication protocol (e.g., HTTPS). The audio data is compressed to transmit the audio file efficiently using less bandwidth.

[0281] Input: Recorded audio data

[0282] Output: Compressed audio data file

[0283] Specifically, the recorded audio data is converted into a compressed format such as .mp3 or .opus and uploaded to a server using the HTTPS protocol.

[0284] Step 3:

[0285] The server passes the received voice data to a voice recognition engine, which converts the voice into text.

[0286] Input: Compressed audio data file

[0287] Output: Text data

[0288] Specifically, the voice data is converted into text data using the speech recognition engines of Google Cloud Speech-to-Text and Amazon Transcribe. The server calls the API to send the audio file and obtains the returned text data.

[0289] Step 4:

[0290] The server analyzes the text data using natural language processing technology to detect specific keywords and phrases.

[0291] Input: Text data

[0292] Output: Analysis results containing keywords and phrases that may be fraudulent

[0293] Specifically, the system analyzes text data using Google Cloud Natural Language API and Amazon Comprehend to detect fraudulent keywords such as "transfer" and "prepaid card."

[0294] Step 5:

[0295] Based on the analysis results, the server sends a push notification to the smartphone, and also notifies the user and their family via email or SMS.

[0296] Input: Suspected fraud analysis results

[0297] Output: Push notification and email / SMS notification

[0298] Specifically, it uses Firebase Cloud Messaging and Amazon SNS to send push notifications to smartphones, as well as email and SMS notifications to users and their families, containing specific details about the suspected scam and a warning message.

[0299] Step 6:

[0300] The server stores the detection results and audio data in a cloud database (such as Firebase Firestore or Amazon DynamoDB) and shares the data with law enforcement agencies if necessary.

[0301] Input: Analysis results and audio data

[0302] Output: Data stored in a cloud database

[0303] Specifically, the system securely transmits and stores voice data and analysis results in a cloud database, and provides authenticated access for data sharing with law enforcement agencies.

[0304] In this way, the individual processing steps work together to effectively operate a system that protects the elderly from fraud.

[0305] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0306] ---

[0307] This invention relates to a system that detects telephone fraud in real time and notifies the elderly person and their family. In particular, by combining it with an emotion engine that recognizes the user's emotions from their voice, the possibility of fraud can be further increased. This system is implemented with the following components.

[0308] This system detects the content of phone conversations in real time, analyzes specific keywords, recognizes the user's emotions, and notifies the elderly and their families.The system consists of the following main components.

[0309] 1. Terminal: A telephone terminal used by the elderly. This terminal records calls and generates audio data.

[0310] 2. Server: Receives voice data, performs speech recognition, generates text data, analyzes keywords and phrases that may be fraudulent, and recognizes user emotions.

[0311] 3. Emotion Engine: Recognizes user emotions from recorded voice data. Emotion recognition analyzes the emotional tone of a call to assess the likelihood of fraud.

[0312] 4. Users: Elderly people and their families. They receive notifications from the server and take appropriate action in case of possible fraud.

[0313] Program processing

[0314] 1. Acquiring audio data

[0315] When the device detects an incoming call, it automatically starts recording the call, so that the conversation between the elderly person and the other party is recorded as digital audio data from the moment the call begins.

[0316] 2. Sending audio data

[0317] After the call ends, the device compresses the recorded audio data to improve data transfer efficiency and save storage space. The compressed audio data is then temporarily stored on the device.

[0318] The device uses a secure protocol to send compressed audio data to the server, using encrypted communications over the internet to ensure data privacy and security.

[0319] 3. Converting voice data to text

[0320] The server passes the received voice data to a voice recognition engine, which converts the voice into text. The voice recognition engine then analyzes the voice data and outputs it as "character string data."

[0321] 4. Natural Language Processing for Fraud Detection

[0322] The server then inputs the generated text data into a natural language processing (NLP) module, which analyzes the text and detects specific keywords and phrases, such as whether it contains terms associated with fraud, such as "transfer" or "prepaid card."

[0323] 5. Emotion recognition

[0324] The server inputs the voice data into the emotion engine and analyzes the user's emotions. The emotion engine recognizes the user's emotions (e.g., anxiety, tension, fear) from the tone, pitch, and intensity of the voice.

[0325] The server integrates the emotion recognition results with the NLP results to provide a comprehensive assessment of the likelihood of fraud.

[0326] 6. Notification Processing

[0327] Based on the analysis results, the server sends a notification to the elderly person and their family, including the specific wording of the suspected fraudulent call and a summary of the call.

[0328] The server then adjusts the notification content based on the emotion recognition results. For example, if the user's emotions are very unstable, the notification will include an alert urging immediate action.

[0329] 7. User Response

[0330] The user receives a notification and checks its contents, which include the suspected fraudulent text, a summary of the call, and the results of emotion recognition. The user can then contact the elderly person and take action if necessary.

[0331] 8. Data Storage and Sharing

[0332] The server stores the analysis results and audio data in a database, which can be used for later review and analysis.

[0333] Where necessary, we may share collected data with law enforcement agencies to help them investigate fraudulent trends and patterns.

[0334] Specific examples

[0335] Detection of bank transfer fraud and emotion recognition

[0336] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "This is my son. I urgently need money, so please transfer it to the specified account right now."

[0337] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[0338] 3. The server passes the voice data to a voice recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer it to the specified account immediately."

[0339] 4. The server analyzes the text data and detects the phrase "Transfer the money to the specified account." It determines that the message is likely to be fraudulent.

[0340] 5. The server passes the voice data to an emotion engine to recognize the user's emotions (e.g., nervousness, anxiety). The recognition results are evaluated as factors that further increase the likelihood of fraud.

[0341] 6. The server immediately sends a notification to the user (family member of the elderly person) saying, "You have received a potentially fraudulent call. The message is, 'We urgently need money, so please transfer it to the specified account immediately.' Please be careful as the user's emotions are unstable."

[0342] 7. The user will receive a notification and contact the elderly person to warn them and instruct them not to make the transfer.

[0343] Prepaid Card Fraud Detection and Emotion Recognition

[0344] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "Please buy a prepaid card at a convenience store and give me the number."

[0345] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[0346] 3. The server passes the voice data to a voice recognition engine, which generates text data such as, "Please purchase a prepaid card at a convenience store and tell me the number."

[0347] 4. The server analyzes the text data and detects the phrase "purchase a prepaid card." It determines that the data is likely fraudulent.

[0348] 5. The server passes the voice data to an emotion engine to recognize the user's emotion (e.g., anxiety, alarm). The recognition result is evaluated as a factor that further increases the likelihood of fraud.

[0349] 6. The server immediately sends a notification to the user (family member of the elderly person) saying, "You have received a potentially fraudulent call. The caller asks you to purchase a prepaid card at a convenience store and provide the number. Please be careful as the user's emotions are unstable."

[0350] 7. The user will receive a notification and contact the elderly person to warn them and provide guidance on how to prevent fraud.

[0351] Through real-time analysis of voice data and emotion recognition, this system is an effective means of protecting the safety of the elderly and can be useful in combating widespread fraud.

[0352] ---

[0353] That's all.

[0354] The processing flow will be explained below.

[0355] ---

[0356] Step 1:

[0357] When the device detects an incoming call, it automatically starts recording the call and continues recording until the call ends, saving the conversation as digital audio data.

[0358] ---

[0359] Step 2:

[0360] After the call ends, the device compresses the recorded audio data, which saves storage space and improves data transmission efficiency.

[0361] ---

[0362] Step 3:

[0363] The device uses a secure protocol to send compressed audio data to the server, where it is encrypted and securely transmitted over the internet.

[0364] ---

[0365] Step 4:

[0366] The server passes the received voice data to a speech recognition engine, which converts the voice into text. The speech recognition engine analyzes the voice data and generates corresponding text data.

[0367] ---

[0368] Step 5:

[0369] The server inputs the generated text data into a natural language processing (NLP) module, which analyzes the text data and detects specific keywords and phrases related to fraud (e.g., "transfer money" or "prepaid card").

[0370] ---

[0371] Step 6:

[0372] The server passes the voice data to the emotion engine to recognize the user's emotion. The emotion engine analyzes the voice tone, pitch, intensity, etc. to identify the user's emotion (e.g., anxiety, tension, fear, etc.).

[0373] ---

[0374] Step 7:

[0375] The server combines the analysis results of the NLP module and the recognition results of the emotion engine to comprehensively evaluate the likelihood of fraud. If the emotion engine's results increase suspicion of fraud, the evaluation is revised upward.

[0376] ---

[0377] Step 8:

[0378] If the server determines that fraud is likely, it prepares a notification message containing the suspected fraudulent text detected, a summary of the call, and the results of emotion recognition.

[0379] ---

[0380] Step 9:

[0381] The server sends a notification message to the user (elderly person and their family) via SMS, email, or a dedicated app.

[0382] ---

[0383] Step 10:

[0384] The user receives and reviews the notification, which includes the specific suspected fraudulent text, a summary of the call, and the results of emotion recognition. The user then contacts the senior and takes appropriate action to address the potential fraud.

[0385] ---

[0386] Step 11:

[0387] The server stores the analysis results and audio data in a database, making it easy to check and analyze later.

[0388] ---

[0389] Step 12:

[0390] If necessary, the server will initiate the process of sharing collected data with law enforcement agencies to investigate fraudulent trends and patterns and assist in legal action.

[0391] ---

[0392] That's all.

[0393] Example 2

[0394] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0395] Conventional telephone fraud prevention systems are limited in their ability to record and analyze call content, making it difficult to detect fraud in real time. Furthermore, they are unable to take into account the emotions of the elderly, such as anxiety and tension, which can lead to reduced accuracy in detecting fraudulent activity. Furthermore, notification functions are inadequate, leaving a lack of means to quickly alert the elderly and their families. There is a need for a new system that can solve these problems.

[0396] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for recording voice, a means for compressing and transmitting the recorded voice, a means for converting the transmitted voice into text using a voice recognition engine, a means for analyzing the generated text using a natural language processing module and detecting specific keywords, a means for recognizing the user's emotions from the voice data, a means for comprehensively evaluating the possibility of fraud, a means for sending a notification based on the evaluation result, and a means for saving the detection result and the voice data. This makes it possible to analyze the content of the call and the user's emotions in real time, identify the possibility of fraud with high accuracy, and promptly notify the elderly person and their family.

[0397] A "voice recording means" is a device capable of recording voice data in digital format at the initiation of a telephone call.

[0398] The "means for compressing and transmitting recorded audio" is a device that has the function of compressing recorded audio data in a manner that reduces the data volume and transmitting it to a server using a secure protocol.

[0399] The "means for converting transmitted voice into text using a voice recognition engine" refers to software or hardware for analyzing received voice data on a server and converting it into corresponding text data.

[0400] "Means for analyzing the generated text using a natural language processing module and detecting specific keywords" refers to a software module that has the function of analyzing text data using natural language processing technology and detecting specific keywords and phrases related to fraud.

[0401] "Means for recognizing user emotions from voice data" refers to a system or software that analyzes the tone, pitch, intensity, etc. of recorded voice data to recognize the user's emotional state (e.g., anxiety, tension, fear).

[0402] "Means for comprehensively assessing the likelihood of fraud" refers to algorithms or systems that integrate the results of keyword detection using natural language processing and emotion recognition to determine the likelihood of fraud.

[0403] The "means for sending notifications based on the evaluation results" refers to a system that has the function of sending notifications containing warnings and detailed information to the elderly person and their family if it is determined that there is a high possibility of fraud.

[0404] The "means for storing the detection results and audio data" refers to a device or system that has the function of storing the analysis results and recorded audio data in storage such as a database for later confirmation or analysis.

[0405] This invention relates to a system that detects telephone fraud in real time and notifies the elderly person and their family. The main purpose of this system is to record voice, analyze the recorded voice, and evaluate the possibility of fraud with high accuracy. In particular, by incorporating a mechanism to recognize the user's emotions from voice data, the accuracy of fraud detection can be further improved. This system is realized using the following specific hardware and software.

[0406] Hardware and Software Configuration

[0407] Terminal

[0408] The telephone terminals used by the elderly have the ability to record phone conversations in real time. The terminals record conversations as digital audio data from the moment the call begins until it ends. The recorded audio data is compressed for efficient storage and sent to a server using a secure protocol (e.g., HTTPS).

[0409] server

[0410] The server is the central location for analyzing the received audio data. It contains the following main software modules:

[0411] Speech recognition engine: Converts received voice data into text data. Google Cloud Speech-to-Text and IBM Watson Speech to Text are available as this engine.

[0412] Natural Language Processing (NLP) module: Analyzes the generated text data and detects specific keywords and phrases related to fraud. This module is powered by IBM Watson Natural Language Understanding.

[0413] Emotion engine: Used to recognize user emotions from recorded voice data. Tools such as Affectiva and Microsoft Azure Emotion Recognition are available.

[0414] Database: Analysis results and audio data are stored. The stored data is used for later review and analysis.

[0415] User

[0416] The users are the elderly themselves and their families. If the server determines that there is a possibility of fraud, the system receives a notification and takes appropriate action.

[0417] Specific examples

[0418] Detection of bank transfer fraud and emotion recognition

[0419] 1. The device detects incoming calls and records the conversation. For example, a scammer might say, "This is my son. I urgently need money, so please transfer it to a specified account right now."

[0420] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[0421] 3. The server passes the voice data to a voice recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer it to the specified account immediately."

[0422] 4. The server analyzes the text data and detects the phrase "Please transfer the money to the specified account." It determines that the data is likely to be fraudulent.

[0423] 5. The server passes the voice data to an emotion engine to recognize the user's emotions (e.g., tension, anxiety). The recognition results are used as factors to further increase the likelihood of fraud.

[0424] 6. The server immediately notifies the user (family member of the elderly person) that "You have received a potentially fraudulent call. The caller says, 'We urgently need money, so please transfer the money to the specified account immediately.' Please be careful as the user's emotions are unstable."

[0425] 7. The user will receive a notification and contact the elderly person to warn them and instruct them not to make the transfer.

[0426] Prepaid Card Fraud Detection and Emotion Recognition

[0427] 1. The device detects an incoming call and records the call. For example, a scammer might ask you to buy a prepaid card at a convenience store and provide the number.

[0428] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[0429] 3. The server passes the voice data to a voice recognition engine, which generates text data such as, "Please purchase a prepaid card at a convenience store and tell me the number."

[0430] 4. The server analyzes the text data and detects the phrase "purchase a prepaid card." It determines that the data is likely fraudulent.

[0431] 5. The server passes the voice data to an emotion engine to recognize the user's emotion (e.g., anxiety, vigilance). The recognition result is used as a factor to further increase the likelihood of fraud.

[0432] 6. The server immediately notifies the user (family member of the elderly person) that "You have received a potentially fraudulent call. The caller asks you to purchase a prepaid card at a convenience store and provide the number. Please be careful as the user's emotions may be unstable."

[0433] 7. The user will receive a notification and contact the elderly person to warn them and provide guidance on how to prevent fraud.

[0434] Through real-time analysis of voice data and emotion recognition, the system is an effective means of protecting the safety of the elderly and can help combat widespread fraud.

[0435] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0436] Step 1:

[0437] When the device detects an incoming call, it automatically starts recording the call. It receives a call start signal as input. Based on this, it converts the voice data during the call into a digital format in real time and temporarily stores the data. Specifically, the recording process starts the moment the phone receiver is lifted, and the display shows "Recording." The output is the recorded digital voice data.

[0438] Step 2:

[0439] When a call ends, the device compresses the recorded voice data. It receives the digital voice data of the ended call as input. Based on this data, it reduces the data size using voice compression techniques such as High Efficiency Audio Codec (HE-AAC). Specifically, the moment the call ends, the message "Compressing voice data..." is displayed. The compressed voice data is temporarily stored on the device. The output is compressed voice data.

[0440] Step 3:

[0441] The device uses a secure protocol (e.g. HTTPS) to send compressed audio data to the server. As input, it receives compressed audio data. Based on this data, it uses encrypted communications to send it to the server while maintaining the privacy and security of the data. Specifically, it displays "Sending data..." and then displays "Transmission complete" once the transmission is complete. The output is the audio data sent to the server.

[0442] Step 4:

[0443] The server passes the received voice data to a voice recognition engine, which converts the voice into text. Compressed voice data is received as input. Based on this, the voice recognition engine (e.g., Google Cloud Speech-to-Text) analyzes the voice data and generates corresponding text data. Specifically, a progress bar appears saying "Analyzing voice data..." as the analysis progresses. The output is the generated text data.

[0444] Step 5:

[0445] The server inputs the generated text data into a natural language processing (NLP) module to detect specific keywords and phrases. As input, it receives text data from a speech recognition engine. This data is analyzed using a natural language processing module (e.g., IBM Watson Natural Language Understanding) to detect keywords related to fraud (e.g., "transfer" and "prepaid card"). Specifically, it displays "Text analysis in progress..." and, if possible fraud is detected, displays "Suspected fraud." The output is a flag or list of the detection results.

[0446] Step 6:

[0447] The server inputs the voice data into an emotion engine to analyze the user's emotions. Recorded voice data is received as input. This data is analyzed by an emotion engine (e.g., Affectiva or Microsoft Azure Emotion Recognition) to recognize the user's emotions (e.g., anxiety, tension, fear) from the tone, pitch, and intensity of the voice. Specifically, the system displays "Analyzing emotions..." and outputs information such as "User emotion: tension." The output is the recognized emotion data.

[0448] Step 7:

[0449] The server integrates the emotion recognition results with the NLP results to comprehensively assess the likelihood of fraud. As input, it receives the detection results of the natural language processing module and the recognition results of the emotion engine. Based on this data, the algorithm evaluates the likelihood of fraud. Specifically, it displays "Evaluating..." and, if the result indicates a high likelihood of fraud, it displays "High likelihood of fraud." The output is a fraud likelihood flag as the evaluation result.

[0450] Step 8:

[0451] The server sends a notification to the elderly person and their family based on the evaluation results. The server receives the fraud possibility evaluation results as input. Based on this data, it generates the notification content and sends it to the elderly person and their family. The specific operation is to display "Sending notification..." and include specific wording of the suspected fraud and a summary of the call. The output is a message indicating that the notification has been sent.

[0452] Step 9:

[0453] The server saves the analysis results and audio data in a database. It receives the evaluation results and audio data as input. Based on this data, it processes the data for storage in the database. Specifically, it displays "Saving data..." and then displays "Saving completed" once the data has been saved. The output is a confirmation message that the data has been saved.

[0454] Step 10:

[0455] The user receives the notification, checks its contents, and contacts the elderly person to take appropriate action. The input is a notification from the server. This notification includes the specific wording of the suspected fraud, a summary of the call, and the results of emotion recognition. The specific action is to check the notification and contact the elderly person to warn them. The output is a message from the user indicating that the action has been taken.

[0456] (Application example 2)

[0457] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0458] Telephone fraud targeting the elderly is increasing year by year, and the methods are becoming more sophisticated. The elderly are particularly vulnerable to methods that create anxiety and fear, making it difficult to prevent fraud damage before it occurs. The purpose of this invention is to provide a system that detects telephone fraud in real time and promptly notifies the elderly and their families, thereby preventing fraud damage.

[0459] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recording voice, means for converting the recorded voice into text, means for analyzing the generated text and detecting specific keywords, means for recognizing emotions from voice data, means for sending notifications based on the detected keywords and emotion recognition results, and means for saving the detection results and voice data. This makes it possible to detect fraud-related keywords and changes in emotions contained in the content of calls in real time and to notify the elderly person and their family members promptly and appropriately.

[0460] A "voice recording device" is a device or component that has the ability to record telephone conversations as digital audio data.

[0461] The "means for converting recorded voice to text" is a voice recognition engine or software that converts recorded voice data into character string data.

[0462] The "means for analyzing the generated text and detecting specific keywords" refers to a natural language processing (NLP) module and analysis algorithms that identify specific terms and phrases within the text data.

[0463] The "means for recognizing emotions from voice data" is an emotion analysis engine that analyzes voice characteristics such as tone, pitch, and intensity to recognize the emotional state of the caller.

[0464] The "means for sending notifications based on detected keywords and emotion recognition results" refers to a communication module or notification system that sends alerts to users and their families in the event of suspected fraud.

[0465] "Means for storing detection results and audio data" refers to a database or storage system that stores the analyzed results and recorded audio data in digital format.

[0466] The present invention relates to a system for monitoring telephone calls in real time and notifying seniors and their families of possible fraud. The system includes a means for recording voice, a means for converting the recorded voice into text, a means for analyzing the generated text and detecting specific keywords, a means for recognizing emotions from the voice data, a means for sending notifications based on the detected keywords and emotion recognition results, and a means for storing the detection results and the voice data.

[0467] A way to record audio

[0468] The telephone terminals used by users are equipped with a function to record calls as digital audio data. This function starts recording from the moment the call starts and saves it when the call ends. A specific example is a built-in application on a smartphone that provides a recording function.

[0469] A means of converting recorded audio into text

[0470] When the call ends, the device sends the recorded voice data to the server, which converts the voice data into text using a speech recognition engine (e.g., Google Speech Recognition API). This saves the call contents as text data.

[0471] A means of analyzing the generated text to detect specific keywords

[0472] The server passes the generated text data to a natural language processing (NLP) module, which is responsible for detecting specific keywords and phrases (e.g., "transfer money" or "prepaid card"). If the NLP module detects keywords that are likely to be fraudulent, the process proceeds to the next step.

[0473] A method for recognizing emotions from voice data

[0474] At the same time, the server analyzes the user's emotions using a sentiment analysis engine (e.g., the sentiment analysis model from the transformers library), which identifies emotions (e.g., anxiety, fear) based on the tone, pitch, and intensity of the voice.

[0475] A means of sending notifications based on detected keywords and emotion recognition results

[0476] The server integrates the detected keywords with the results of emotion recognition, and if there is a high possibility of fraud, it sends a notification to the user and their family. The notification includes the specific content of the call and the detected emotion, and contains information urging the user to take prompt action. Notifications can be sent via email or push notification.

[0477] A means of storing detection results and audio data

[0478] The server stores all analysis results and audio data in a database, which can then be reviewed and shared with law enforcement agencies.

[0479] Specific examples

[0480] For example, if an elderly person receives a call from a scammer saying, "I want to buy a prepaid card at a convenience store and please tell me the number," the following process takes place: The call is recorded and converted into text using speech recognition. Next, the NLP module detects the phrase "I want to buy a prepaid card." At the same time, the sentiment analysis engine recognizes the user's anxiety. As a result, the server determines that there is a high possibility of fraud and sends a notification to the user's family saying, "You have received a potentially fraudulent call. The call says, 'I want to buy a prepaid card at a convenience store and please tell me the number.' Please be careful as the user is in an emotionally unstable state."

[0481] Prompt Sentence Examples

[0482] "Perform the following tasks to generate text from the audio data containing keywords that increase suspicion of fraud and detect the user's emotional state (anxiety, fear, etc.). The transcript of the audio data is as follows: "This is my son. I'd like to buy a prepaid card at a convenience store and would like the number." Based on this, assess the likelihood of fraud and the emotional state."

[0483] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0484] Step 1:

[0485] The device detects an incoming call and activates the call recording function. When the call starts, the device automatically starts recording the conversation as digital audio data. The input is the incoming call signal, and the output is the recorded audio data.

[0486] Step 2:

[0487] When the call ends, the device compresses the recorded voice data and sends it to the server. The device transmits the data using a secure protocol over the Internet, maintaining the privacy and security of the data. The input is the recorded voice data, and the output is a data packet containing the compressed voice data.

[0488] Step 3:

[0489] The server passes the received voice data to a speech recognition engine, which converts the voice into text. The speech recognition engine (for example, Google Speech Recognition API) analyzes the voice data and generates corresponding text data. The input is compressed voice data, and the output is text data.

[0490] Step 4:

[0491] The server inputs the generated text data into a natural language processing (NLP) module, which analyzes the text to detect specific keywords and phrases. The NLP module searches for fraud-related keywords (e.g., "transfer" or "prepaid card") within the text and, if detected, proceeds to the next step. The input is the text data, and the output is the detected keyword information.

[0492] Step 5:

[0493] The server inputs the voice data into an emotion recognition engine to analyze the user's emotional state. The emotion recognition engine (e.g., the emotion analysis model in the transformers library) analyzes voice features such as tone, pitch, and intensity to identify the user's emotion (e.g., anxiety, fear). The input is the voice data, and the output is the emotion recognition result.

[0494] Step 6:

[0495] The server integrates the detected keywords and emotion recognition results to comprehensively assess the likelihood of fraud. Based on this result, if there is a high suspicion of fraud, it prepares to send an alert to the user and their family. The input is the keyword detection information and emotion recognition results, and the output is a notification alert message.

[0496] Step 7:

[0497] The server constructs an alert message and sends it to the specified notification recipients (e.g., the user's family). Notification methods include email and push notifications. The input is the alert message for notification, and the output is the sent notification.

[0498] Step 8:

[0499] The server stores all analysis results and audio recordings in a database, allowing them to be reviewed at a later date and shared with law enforcement if necessary. The inputs are analysis results and audio recordings, and the output is the stored data.

[0500] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0501] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0502] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0503] [Second embodiment]

[0504] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0505] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0506] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0507] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0508] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0509] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0510] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0511] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0512] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0513] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0514] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0515] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0516] ---

[0517] The present invention relates to a system for detecting telephone fraud in real time and notifying elderly people and their families. A specific embodiment of the system and program processing will be described below.

[0518] This system detects the contents of phone calls in real time, analyzes specific keywords, and notifies the elderly and their families. The system consists of the following main components:

[0519] 1. Terminal: A telephone terminal used by the elderly. This terminal records calls and generates audio data.

[0520] 2. Server: Receives voice data, performs speech recognition, generates text data, and analyzes and detects keywords and phrases that may be fraudulent.

[0521] 3. Users: Elderly people and their families. They receive notifications from the server and take appropriate action in case of possible fraud.

[0522] Program processing

[0523] 1. Acquiring audio data

[0524] When the device detects an incoming call, it automatically starts recording the call, and when the call ends, it temporarily saves the recorded audio data.

[0525] 2. Sending audio data

[0526] The device compresses the recorded audio data and sends it to the server using a secure protocol.

[0527] 3. Converting voice data to text

[0528] The server passes the received voice data to a voice recognition engine, converts the voice into text, and obtains the text data generated by the voice recognition engine.

[0529] 4. Natural Language Processing for Fraud Detection

[0530] The server analyzes the text data, detecting specific keywords and phrases (e.g., "transfer" or "prepaid card"), and uses natural language processing techniques to evaluate the content for potential fraud.

[0531] 5. Notification Processing

[0532] Based on the analysis results, the server sends a notification to the elderly person and their family, including the specific wording of the suspected fraudulent call and a summary of the call.

[0533] Users can receive notifications, review the details, and take immediate action, for example, allowing family members to contact the elderly and prevent fraud before it occurs.

[0534] 6. Data Storage and Sharing

[0535] The server stores the detection results and audio data in a database, allowing for detailed analysis and review later if necessary.

[0536] Where necessary, we may share collected data with law enforcement agencies to help them investigate fraud trends and patterns.

[0537] Specific examples

[0538] Detection of bank transfer fraud

[0539] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "This is my son. I urgently need money, so please transfer it to the specified account right now."

[0540] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[0541] 3. The server passes the voice data to a voice recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer it to the specified account immediately."

[0542] 4. The server analyzes the text data and detects the phrase "Transfer the money to the specified account." It determines that the message is likely to be fraudulent.

[0543] 5. The server immediately sends a notification to the user (the elderly person's family member) saying, "You have received a potentially fraudulent call. The message is, 'We urgently need money, so please transfer it to the specified account immediately.' Please be careful."

[0544] 6. The user will receive a notification and contact the elderly person to warn them and instruct them not to make the transfer.

[0545] Prepaid card fraud detection

[0546] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "Please buy a prepaid card at a convenience store and give me the number."

[0547] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[0548] 3. The server passes the voice data to a voice recognition engine, which generates text data such as, "Please purchase a prepaid card at a convenience store and tell me the number."

[0549] 4. The server analyzes the text data and detects the phrase "purchase a prepaid card." It determines that the data is likely fraudulent.

[0550] 5. The server immediately sends a notification to the user (the elderly person's family member) saying, "You have received a potentially fraudulent call. The caller asks you to purchase a prepaid card at a convenience store and provide the number. Please be careful."

[0551] 6. The user will receive a notification and contact the elderly person to warn them and provide guidance on how to prevent fraud.

[0552] Through real-time analysis and notification of voice data, this system is an effective means of protecting the safety of the elderly and is useful in combating widespread fraud.

[0553] ---

[0554] That's all.

[0555] The processing flow will be explained below.

[0556] ---

[0557] Step 1:

[0558] When the device detects an incoming call, it automatically starts recording the call, so that the conversation between the elderly person and the other party is recorded as digital audio data from the moment the call begins.

[0559] ---

[0560] Step 2:

[0561] After the call ends, the device compresses the recorded audio data to improve data transfer efficiency and save storage space. The compressed audio data is then temporarily stored on the device.

[0562] ---

[0563] Step 3:

[0564] The device uses a secure protocol to send compressed audio data to the server, using encrypted communications over the internet to ensure data privacy and security.

[0565] ---

[0566] Step 4:

[0567] The server passes the received voice data to a voice recognition engine, which converts the voice into text. The voice recognition engine then analyzes the voice data and outputs it as "character string data."

[0568] ---

[0569] Step 5:

[0570] The server then inputs the generated text data into a natural language processing (NLP) module, which analyzes the text and detects specific keywords and phrases, such as whether it contains terms associated with fraud, such as "transfer" or "prepaid card."

[0571] ---

[0572] Step 6:

[0573] The server evaluates the analysis results from the NLP module and, if it determines that there is a high possibility of fraud, begins preparing a notification. At this stage, the notification content is generated and a message is created to be sent to the elderly person and their family.

[0574] ---

[0575] Step 7:

[0576] The server sends a notification message, which can be sent to the user via SMS, email, or a dedicated app, allowing the user to quickly become aware of potential fraud and take action.

[0577] ---

[0578] Step 8:

[0579] The user receives a notification, which includes the suspected fraudulent text and a summary of the call, and can then contact the senior to provide guidance or advice.

[0580] ---

[0581] Step 9:

[0582] The server stores the analysis results and audio data in a database, which can be used for later review and analysis.

[0583] ---

[0584] Step 10:

[0585] If necessary, the server will initiate the process of sharing collected data with law enforcement agencies to investigate fraud trends and patterns and assist in legal action.

[0586] ---

[0587] That's all.

[0588] Example 1

[0589] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0590] In recent years, telephone fraud has been on the rise, with elderly people being particularly vulnerable. To prevent fraud, it is necessary to detect suspicious calls in real time and promptly notify the elderly person and their family. However, conventional systems are unable to adequately perform this real-time detection and notification, which often results in damage occurring. For this reason, there is a need for a system that can detect and notify fraud more effectively and quickly.

[0591] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0592] In this invention, the server includes a means for compressing voice data and transmitting it using a secure protocol, a means for converting recorded voice into text, a natural language processing means for analyzing the generated text and detecting specific keywords, a means for sending a notification based on the detected keywords, a means for sending the notification to the user, a means for saving the analysis results and the voice data, and a means for sharing the analysis results and the voice data with law enforcement agencies as needed. This makes it possible to detect suspected fraudulent calls in real time and quickly notify the elderly person and their family.

[0593] "Audio recording means" refers to a device or program that automatically records telephone conversations and saves the audio data.

[0594] "Means for compressing recorded audio data and transmitting it using a secure protocol" refers to a device or program that compresses recorded audio files using a compression algorithm and transmits them to a server using a secure communication protocol such as HTTPS or SSL.

[0595] A "means for converting recorded speech to text" is a system or program that uses a speech recognition engine to analyze speech data and convert it into a human-understandable text format.

[0596] "Natural language processing means for analyzing generated text and detecting specific keywords" means a system or program that uses natural language processing techniques to analyze text data and detect specific keywords or phrases that may be suspected of being fraudulent.

[0597] "Means for sending notifications based on detected keywords" means a system or program for sending warnings or notifications to users when detected keywords or phrases indicate suspected fraud.

[0598] "Means for sending notifications to users" means a system or program for sending notifications to the elderly person and their family members via SMS, email, or other communication means.

[0599] "Means for storing analysis results and audio data" refers to a system or program for storing analysis results and audio data in a database or file system, and keeping them in a state where they can be accessed and analyzed later.

[0600] "Means for sharing analytics results and audio data with law enforcement as needed" means a system or program that shares stored analytics results and audio data with law enforcement to assist in fraud investigation and prevention.

[0601] "Communication terminals used by the elderly" refer to telephones, smartphones, or other communication devices capable of making calls that are primarily used by the elderly in their daily lives.

[0602] A "server with a speech recognition engine" is a server that has speech recognition capabilities to convert received speech data into text, and is a device that has the computing resources to execute programs and algorithms to analyze speech.

[0603] The present invention relates to a system for detecting telephone fraud in real time and notifying elderly people and their families of the fraud. Specific embodiments of the system will be described below.

[0604] This system consists of the following main hardware and software:

[0605] Hardware and Software Configuration

[0606] 1. Device:

[0607] Telephones and smartphones are used as communication devices by the elderly. These devices record calls and generate audio data.

[0608] The software used is a module for recording (for example, a recording class developed in Java or Swift).

[0609] 2. Server:

[0610] A server that receives voice data, performs speech recognition, and generates text data. The server is equipped with a speech recognition engine (e.g., Google Speech-to-Text API) and a natural language processing engine (e.g., SpaCy or NLTK).

[0611] To run the program, web frameworks such as Python, Flask, and Django are used.

[0612] 3. User:

[0613] The elderly person and their family members receive notifications from the server and take appropriate action if there is a possibility of fraud.

[0614] Data processing and calculation

[0615] Acquiring audio data

[0616] When the device detects an incoming call, it automatically starts recording the call. For example, you can use the smartphone's built-in app or a dedicated recording app. When the call ends, the recorded audio data is temporarily stored on the device.

[0617] Sending audio data

[0618] The device compresses the recorded audio data and sends it to the server using a secure protocol (e.g., HTTPS or SSL) using an algorithm such as GZIP.

[0619] Converting audio data to text

[0620] The server passes the received voice data to a speech recognition engine (e.g., Google Speech-to-Text API) and converts the voice into text, which is then stored in a database.

[0621] Natural Language Processing for Fraud Detection

[0622] The server analyzes the text data using a natural language processing engine (e.g., SpaCy or NLTK) to detect specific keywords and phrases (e.g., "transfer," "prepaid card," etc.).

[0623] Notification Processing

[0624] Based on the analysis results, the server sends a notification to the elderly person and their family, including specific wording of the suspected fraud and a summary of the call. For example, notifications are sent via SMS or email using Twilio or SendGrid.

[0625] Data storage and sharing

[0626] The server stores the detection results and audio data in a database using MySQL or PostgreSQL. The stored data can be analyzed and reviewed in detail at a later date, and can be shared with law enforcement agencies if necessary.

[0627] Specific examples

[0628] Detection of bank transfer fraud

[0629] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "This is my son. I urgently need money, so please transfer it to the specified account right now."

[0630] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[0631] 3. The server passes the voice data to a voice recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer it to the specified account immediately."

[0632] 4. The server analyzes the text data and detects the phrase "Transfer the money to the specified account." It determines that the message is likely to be fraudulent.

[0633] 5. The server immediately sends a notification to the user (the elderly person's family member) saying, "You have received a potentially fraudulent call. The message is, 'We urgently need money, so please transfer it to the specified account immediately.' Please be careful."

[0634] 6. The user receives a notification and contacts the elderly person to warn them and prevent any damage from occurring.

[0635] Prepaid card fraud detection

[0636] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "Please buy a prepaid card at a convenience store and give me the number."

[0637] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[0638] 3. The server passes the voice data to a voice recognition engine, which generates text data such as, "Please purchase a prepaid card at a convenience store and tell me the number."

[0639] 4. The server analyzes the text data and detects the phrase "purchase a prepaid card." It determines that the data is likely fraudulent.

[0640] 5. The server immediately sends a notification to the user (the elderly person's family member) saying, "You have received a potentially fraudulent call. The caller asks you to purchase a prepaid card at a convenience store and provide the number. Please be careful."

[0641] 6. The user receives a notification and contacts the elderly person to warn them and prevent any damage from occurring.

[0642] Prompt Sentence Examples

[0643] In the case of detecting bank transfer fraud

[0644] "Please tell me how to build a system that notifies the elderly person and their family when they receive a call from someone pretending to be their son requesting that they transfer money to a specified account."

[0645] Prepaid card fraud detection

[0646] "Please tell me how to build a system that notifies elderly people and their families when they purchase a prepaid card at a convenience store and are asked to provide the card number."

[0647] The system provides real-time voice analysis and rapid notification to protect against fraud targeting seniors, making it effective in combating a wide range of frauds.

[0648] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0649] Step 1: Acquire audio data

[0650] When the device detects an incoming call, it automatically starts recording the call, recording the contents of the call in real time.

[0651] Input: Call audio

[0652] Data processing: Save call audio as a recording file

[0653] Output: Recording file

[0654] How it works: The device's calling app triggers an incoming call event, calls a recording module (for example, a recording class written in Java or Swift) to start recording, and then saves the recording file in the device's experience data storage after the call ends.

[0655] Step 2: Sending audio data

[0656] The device compresses the recorded audio data and sends it to the server using a secure protocol.

[0657] Input: Recording file

[0658] Data processing: Compress the audio data using GZIP or other methods and package it for HTTPS requests

[0659] Output: Compressed audio data is sent to the server

[0660] What it does: It compresses the recorded audio data using the GZIP compression algorithm and sends it to the server's API endpoint using the HTTPS protocol, for example, using Python's requests library or Java's HttpURLConnection class.

[0661] Step 3: Convert audio data to text

[0662] The server passes the received voice data to a voice recognition engine, which converts the voice into text.

[0663] Input: Compressed audio data

[0664] Data processing: Converting speech to text using a speech recognition engine (such as Google Speech-to-Text API)

[0665] Output: Text data

[0666] What happens: The server receives the audio data, decompresses it, and passes it to the speech recognition engine. The speech recognition engine analyzes the audio file and generates text data, which is then stored in a database.

[0667] Step 4: Natural Language Processing for Fraud Detection

[0668] The server analyzes the text data using a natural language processing engine to detect specific keywords and phrases.

[0669] Input: Text data

[0670] Data processing: Use natural language processing engines (such as SpaCy or NLTK) to extract and analyze specific keywords and phrases within text data.

[0671] Output: Analysis results showing suspected fraudulent phrases and keywords

[0672] How it works: The server passes the text data to a natural language processing engine, which analyzes it for potentially fraudulent keywords and phrases. Based on the analysis, any data that is suspected to be fraudulent is flagged and stored in a database.

[0673] Step 5: Notification Processing

[0674] Based on the analysis results, the server sends a notification to the elderly person and their family.

[0675] Input: Analysis results

[0676] Data processing: Generate notification content and send it using a notification service such as Twilio or SendGrid

[0677] Output: Email and SMS notifications sent to the elderly person and their family

[0678] Specific operation: The server composes the analysis results in JSON format and sends emails or SMS to the elderly person and their family members through existing notification services such as Twilio and SendGrid.

[0679] Step 6: Data storage and sharing

[0680] The server stores the detection results and audio data in a database, which may be shared with law enforcement agencies if necessary.

[0681] Input: Analysis results and audio data

[0682] Data processing: Analysis results and audio data are written to a database and shared with law enforcement agencies as needed

[0683] Output: Analysis results and audio data stored in a database

[0684] What it does: The server uses an ORM (Object-Relational Mapping) tool such as SQLAlchemy to write the analysis results and audio data to a database (such as MySQL or PostgreSQL), and provides access and data sharing to law enforcement agencies as needed.

[0685] These steps will allow the entire system to work together to protect seniors from phone scams.

[0686] (Application example 1)

[0687] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0688] In recent years, frauds targeting the elderly have been increasing. However, many elderly people do not have sufficient means to avoid fraud, and the damage caused by these frauds is becoming more serious. To solve this problem, a system is needed that can detect telephone frauds in real time and quickly notify the elderly and their families.

[0689] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0690] In this invention, the server includes a means for transmitting recorded voice data to a remote server using a security protocol, a means for converting the voice data into text using a voice recognition engine, a means for analyzing the text data using natural language processing technology, and a means for sending a push notification to a smartphone, thereby enabling real-time detection of suspected fraud and prompt notification to the elderly person and their family.

[0691] "Audio recording means" means a device or function that records the contents of a telephone conversation as digital audio data.

[0692] "Means for converting recorded voice into text" refers to a technology for converting voice data into text in a natural language, i.e., a device or function that generates text data from voice using a voice recognition engine.

[0693] "Means for analyzing the generated text and detecting specific keywords" refers to a device or function that analyzes the generated text data using algorithms or natural language processing techniques to identify keywords or phrases that may be fraudulent.

[0694] A "means for sending notifications based on detected keywords" is a device or function that, when a keyword that indicates suspected fraud is detected, notifies the user or their family of the potential fraud via a notification mechanism.

[0695] "Means for storing detection results and audio data" refers to a device or function that stores the analyzed text data and original audio data in a database or storage system so that they can be accessed later.

[0696] "Means for transmitting recorded audio data to a remote server using a secure protocol" refers to a device or function that compresses the recorded audio data and transfers it to a remote server using a secure communication protocol (e.g., HTTPS).

[0697] "Means for converting voice data into text using a voice recognition engine" refers to a device or function that mechanically converts voice data into text using voice recognition software or a cloud service.

[0698] A "means for analyzing text data using natural language processing techniques" is a device or function that uses natural language processing algorithms or models to evaluate text data and identify keywords or phrases that may contain indicators of fraud.

[0699] A "means for sending push notifications to a smartphone" is a device or system with push notification functionality for quickly notifying users and their families when potential fraud is detected.

[0700] The present invention relates to a system for detecting telephone fraud in real time and notifying elderly people and their families promptly. The following describes a specific embodiment and the hardware and software used.

[0701] System configuration

[0702] The system consists of the following main components:

[0703] 1. Device:

[0704] This is a telephone terminal used by elderly people. It records calls and generates audio data.

[0705] This includes communication devices such as smartphones with call recording functionality installed.

[0706] 2. Server:

[0707] Voice data is received, voice recognition is performed, and text data is generated.

[0708] Specific keywords are analyzed using natural language processing technology.

[0709] Google Cloud Speech-to-Text and Amazon Transcribe are used as speech recognition engines.

[0710] Natural language processing technologies used include Google Cloud Natural Language API and Amazon Comprehend.

[0711] 3. User:

[0712] The elderly themselves and their families.

[0713] Receive notifications from the server and take appropriate action in case of potential fraud.

[0714] About program processing

[0715] The processing flow by the system is shown below.

[0716] 1. Acquire audio data:

[0717] When the device detects an incoming call, it automatically records the call. When the call ends, it stops recording and temporarily saves the audio data.

[0718] 2. Sending audio data:

[0719] The device compresses the recorded audio data and sends it to the server using a secure communication protocol (e.g., HTTPS).

[0720] 3. Audio to text conversion:

[0721] The server passes the received voice data to a speech recognition engine, converts the voice into text data, and generates highly accurate text data from the voice data using Google Cloud Speech-to-Text or Amazon Transcribe.

[0722] 4. Natural Language Processing for Fraud Detection:

[0723] The server analyzes the text data using natural language processing technology (Google Cloud Natural Language API and Amazon Comprehend) to detect keywords and phrases that may be fraudulent.

[0724] 5. Notification Processing:

[0725] Based on the analysis results, the server sends a push notification to the smartphone, and also notifies the user and their family members via email and SMS.

[0726] Use Firebase Cloud Messaging or Amazon SNS for push notification.

[0727] 6. Data Storage and Sharing:

[0728] The server stores the analysis results and audio data in a cloud database (Firebase Firestore or Amazon DynamoDB), which allows the information to be shared with law enforcement agencies if necessary.

[0729] Specific examples

[0730] Example of detecting bank transfer fraud

[0731] The device detects an incoming call and records the conversation. The scammer says, "This is my son. I urgently need money, so please transfer the money to my specified account now." When the call ends, the recorded audio data is compressed and sent to the server. The server passes the audio data to a speech recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer the money to my specified account now." The server analyzes the text data and detects the phrase "transfer the money to my specified account." It immediately sends a notification to the user saying, "You have received a potentially fraudulent call. The message is, 'I urgently need money, so please transfer the money to my specified account now.' Please be careful."

[0732] Prepaid card fraud detection example

[0733] The device detects an incoming call and records the conversation. The scammer says, "Please buy a prepaid card at a convenience store and give me the number." When the call ends, the recorded audio data is compressed and sent to the server. The server passes the audio data to a speech recognition engine, which generates text data saying, "Please buy a prepaid card at a convenience store and give me the number." The server analyzes the text data and detects the phrase "please buy a prepaid card." It immediately sends a notification to the user saying, "You have received a potentially fraudulent call. The message is, 'Please buy a prepaid card at a convenience store and give me the number.' Please be careful."

[0734] In this way, this system can protect the elderly from fraud through real-time analysis and notification of voice data.

[0735] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0736] Step 1:

[0737] The device detects an incoming call and automatically starts recording the call. When the call ends, the recording stops and the audio data is temporarily saved.

[0738] Specifically, the call recording function installed on the smartphone is used to save the audio data during the call in a digital format, such as a .wav or .3gp file.

[0739] Step 2:

[0740] The device compresses the recorded audio data and sends it to the server using a secure communication protocol (e.g., HTTPS). The audio data is compressed to transmit the audio file efficiently using less bandwidth.

[0741] Input: Recorded audio data

[0742] Output: Compressed audio data file

[0743] Specifically, the recorded audio data is converted into a compressed format such as .mp3 or .opus and uploaded to a server using the HTTPS protocol.

[0744] Step 3:

[0745] The server passes the received voice data to a voice recognition engine, which converts the voice into text.

[0746] Input: Compressed audio data file

[0747] Output: Text data

[0748] Specifically, the voice data is converted into text data using the speech recognition engines of Google Cloud Speech-to-Text and Amazon Transcribe. The server calls the API to send the audio file and obtains the returned text data.

[0749] Step 4:

[0750] The server analyzes the text data using natural language processing technology to detect specific keywords and phrases.

[0751] Input: Text data

[0752] Output: Analysis results containing keywords and phrases that may be fraudulent

[0753] Specifically, the system analyzes text data using Google Cloud Natural Language API and Amazon Comprehend to detect fraudulent keywords such as "transfer" and "prepaid card."

[0754] Step 5:

[0755] Based on the analysis results, the server sends a push notification to the smartphone, and also notifies the user and their family via email or SMS.

[0756] Input: Suspected fraud analysis results

[0757] Output: Push notification and email / SMS notification

[0758] Specifically, it uses Firebase Cloud Messaging and Amazon SNS to send push notifications to smartphones, as well as email and SMS notifications to users and their families, containing specific details about the suspected scam and a warning message.

[0759] Step 6:

[0760] The server stores the detection results and audio data in a cloud database (such as Firebase Firestore or Amazon DynamoDB) and shares the data with law enforcement agencies if necessary.

[0761] Input: Analysis results and audio data

[0762] Output: Data stored in a cloud database

[0763] Specifically, the system securely transmits and stores voice data and analysis results in a cloud database, and provides authenticated access for data sharing with law enforcement agencies.

[0764] In this way, the individual processing steps work together to effectively operate a system that protects the elderly from fraud.

[0765] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0766] ---

[0767] This invention relates to a system that detects telephone fraud in real time and notifies the elderly person and their family. In particular, by combining it with an emotion engine that recognizes the user's emotions from their voice, the possibility of fraud can be further increased. This system is implemented with the following components.

[0768] This system detects the content of phone conversations in real time, analyzes specific keywords, recognizes the user's emotions, and notifies the elderly and their families.The system consists of the following main components.

[0769] 1. Terminal: A telephone terminal used by the elderly. This terminal records calls and generates audio data.

[0770] 2. Server: Receives voice data, performs speech recognition, generates text data, analyzes keywords and phrases that may be fraudulent, and recognizes user emotions.

[0771] 3. Emotion Engine: Recognizes user emotions from recorded voice data. Emotion recognition analyzes the emotional tone of a call to assess the likelihood of fraud.

[0772] 4. Users: Elderly people and their families. They receive notifications from the server and take appropriate action in case of possible fraud.

[0773] Program processing

[0774] 1. Acquiring audio data

[0775] When the device detects an incoming call, it automatically starts recording the call, so that the conversation between the elderly person and the other party is recorded as digital audio data from the moment the call begins.

[0776] 2. Sending audio data

[0777] After the call ends, the device compresses the recorded audio data to improve data transfer efficiency and save storage space. The compressed audio data is then temporarily stored on the device.

[0778] The device uses a secure protocol to send compressed audio data to the server, using encrypted communications over the internet to ensure data privacy and security.

[0779] 3. Converting voice data to text

[0780] The server passes the received voice data to a voice recognition engine, which converts the voice into text. The voice recognition engine then analyzes the voice data and outputs it as "character string data."

[0781] 4. Natural Language Processing for Fraud Detection

[0782] The server then inputs the generated text data into a natural language processing (NLP) module, which analyzes the text and detects specific keywords and phrases, such as whether it contains terms associated with fraud, such as "transfer" or "prepaid card."

[0783] 5. Emotion recognition

[0784] The server inputs the voice data into the emotion engine and analyzes the user's emotions. The emotion engine recognizes the user's emotions (e.g., anxiety, tension, fear) from the tone, pitch, and intensity of the voice.

[0785] The server integrates the emotion recognition results with the NLP results to provide a comprehensive assessment of the likelihood of fraud.

[0786] 6. Notification Processing

[0787] Based on the analysis results, the server sends a notification to the elderly person and their family, including the specific wording of the suspected fraudulent call and a summary of the call.

[0788] The server then adjusts the notification content based on the results of emotion recognition. For example, if the user's emotions are very unstable, the notification will include an alert urging immediate action.

[0789] 7. User Response

[0790] The user receives a notification and checks its contents, which include the suspected fraudulent text, a summary of the call, and the results of emotion recognition. The user can then contact the elderly person and take action if necessary.

[0791] 8. Data Storage and Sharing

[0792] The server stores the analysis results and audio data in a database, which can be used for later review and analysis.

[0793] Where necessary, we may share collected data with law enforcement agencies to help them investigate fraudulent trends and patterns.

[0794] Specific examples

[0795] Detection of bank transfer fraud and emotion recognition

[0796] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "This is my son. I urgently need money, so please transfer it to the specified account right now."

[0797] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[0798] 3. The server passes the voice data to a voice recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer it to the specified account immediately."

[0799] 4. The server analyzes the text data and detects the phrase "Transfer the money to the specified account." It determines that the message is likely to be fraudulent.

[0800] 5. The server passes the voice data to an emotion engine to recognize the user's emotions (e.g., nervousness, anxiety). The recognition results are evaluated as factors that further increase the likelihood of fraud.

[0801] 6. The server immediately sends a notification to the user (family member of the elderly person) saying, "You have received a potentially fraudulent call. The message is, 'We urgently need money, so please transfer it to the specified account immediately.' Please be careful as the user's emotions are unstable."

[0802] 7. The user will receive a notification and contact the elderly person to warn them and instruct them not to make the transfer.

[0803] Prepaid Card Fraud Detection and Emotion Recognition

[0804] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "Please buy a prepaid card at a convenience store and give me the number."

[0805] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[0806] 3. The server passes the voice data to a voice recognition engine, which generates text data such as, "Please purchase a prepaid card at a convenience store and tell me the number."

[0807] 4. The server analyzes the text data and detects the phrase "purchase a prepaid card." It determines that the data is likely fraudulent.

[0808] 5. The server passes the voice data to an emotion engine to recognize the user's emotion (e.g., anxiety, alarm). The recognition result is evaluated as a factor that further increases the likelihood of fraud.

[0809] 6. The server immediately sends a notification to the user (family member of the elderly person) saying, "You have received a potentially fraudulent call. The caller asks you to purchase a prepaid card at a convenience store and provide the number. Please be careful as the user's emotions are unstable."

[0810] 7. The user will receive a notification and contact the elderly person to warn them and provide guidance on how to prevent fraud.

[0811] Through real-time analysis of voice data and emotion recognition, this system is an effective means of protecting the safety of the elderly and can be useful in combating widespread fraud.

[0812] ---

[0813] That's all.

[0814] The processing flow will be explained below.

[0815] ---

[0816] Step 1:

[0817] When the device detects an incoming call, it automatically starts recording the call and continues recording until the call ends, saving the conversation as digital audio data.

[0818] ---

[0819] Step 2:

[0820] After the call ends, the device compresses the recorded audio data, which saves storage space and improves data transmission efficiency.

[0821] ---

[0822] Step 3:

[0823] The device uses a secure protocol to send compressed audio data to the server, where it is encrypted and securely transmitted over the internet.

[0824] ---

[0825] Step 4:

[0826] The server passes the received voice data to a speech recognition engine, which converts the voice into text. The speech recognition engine analyzes the voice data and generates corresponding text data.

[0827] ---

[0828] Step 5:

[0829] The server inputs the generated text data into a natural language processing (NLP) module, which analyzes the text data and detects specific keywords and phrases related to fraud (e.g., "transfer money" or "prepaid card").

[0830] ---

[0831] Step 6:

[0832] The server passes the voice data to the emotion engine to recognize the user's emotion. The emotion engine analyzes the voice tone, pitch, intensity, etc. to identify the user's emotion (e.g., anxiety, tension, fear, etc.).

[0833] ---

[0834] Step 7:

[0835] The server combines the analysis results of the NLP module and the recognition results of the emotion engine to comprehensively evaluate the likelihood of fraud. If the emotion engine's results increase suspicion of fraud, the evaluation is revised upward.

[0836] ---

[0837] Step 8:

[0838] If the server determines that fraud is likely, it prepares a notification message containing the suspected fraudulent text detected, a summary of the call, and the results of emotion recognition.

[0839] ---

[0840] Step 9:

[0841] The server sends a notification message to the user (elderly person and their family) via SMS, email, or a dedicated app.

[0842] ---

[0843] Step 10:

[0844] The user receives and reviews the notification, which includes the specific suspected fraudulent text, a summary of the call, and the results of emotion recognition. The user then contacts the senior and takes appropriate action to address the potential fraud.

[0845] ---

[0846] Step 11:

[0847] The server stores the analysis results and audio data in a database, making it easy to check and analyze later.

[0848] ---

[0849] Step 12:

[0850] If necessary, the server will initiate the process of sharing collected data with law enforcement agencies to investigate fraudulent trends and patterns and assist in legal action.

[0851] ---

[0852] That's all.

[0853] Example 2

[0854] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0855] Conventional telephone fraud prevention systems are limited in their ability to record and analyze call content, making it difficult to detect fraud in real time. Furthermore, they are unable to take into account the emotions of the elderly, such as anxiety and tension, which can lead to reduced accuracy in detecting fraudulent activity. Furthermore, notification functions are inadequate, leaving a lack of means to quickly alert the elderly and their families. There is a need for a new system that can solve these problems.

[0856] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for recording voice, a means for compressing and transmitting the recorded voice, a means for converting the transmitted voice into text using a voice recognition engine, a means for analyzing the generated text using a natural language processing module and detecting specific keywords, a means for recognizing the user's emotions from the voice data, a means for comprehensively evaluating the possibility of fraud, a means for sending a notification based on the evaluation result, and a means for saving the detection result and the voice data. This makes it possible to analyze the content of the call and the user's emotions in real time, identify the possibility of fraud with high accuracy, and promptly notify the elderly person and their family.

[0857] A "voice recording means" is a device capable of recording voice data in digital format at the initiation of a telephone call.

[0858] The "means for compressing and transmitting recorded audio" is a device that has the function of compressing recorded audio data in a manner that reduces the data volume and transmitting it to a server using a secure protocol.

[0859] The "means for converting transmitted voice into text using a voice recognition engine" refers to software or hardware for analyzing received voice data on a server and converting it into corresponding text data.

[0860] "Means for analyzing the generated text using a natural language processing module and detecting specific keywords" refers to a software module that has the function of analyzing text data using natural language processing technology and detecting specific keywords and phrases related to fraud.

[0861] "Means for recognizing user emotions from voice data" refers to a system or software that analyzes the tone, pitch, intensity, etc. of recorded voice data to recognize the user's emotional state (e.g., anxiety, tension, fear).

[0862] "Means for comprehensively assessing the likelihood of fraud" refers to algorithms or systems that integrate the results of keyword detection using natural language processing and emotion recognition to determine the likelihood of fraud.

[0863] The "means for sending notifications based on the evaluation results" refers to a system that has the function of sending notifications containing warnings and detailed information to the elderly person and their family if it is determined that there is a high possibility of fraud.

[0864] The "means for storing the detection results and audio data" refers to a device or system that has the function of storing the analysis results and recorded audio data in storage such as a database for later confirmation or analysis.

[0865] This invention relates to a system that detects telephone fraud in real time and notifies the elderly person and their family. The main purpose of this system is to record voice, analyze the recorded voice, and evaluate the possibility of fraud with high accuracy. In particular, by incorporating a mechanism to recognize the user's emotions from voice data, the accuracy of fraud detection can be further improved. This system is realized using the following specific hardware and software.

[0866] Hardware and Software Configuration

[0867] Terminal

[0868] The telephone terminals used by the elderly have the ability to record phone conversations in real time. The terminals record conversations as digital audio data from the moment the call begins until it ends. The recorded audio data is compressed for efficient storage and sent to a server using a secure protocol (e.g., HTTPS).

[0869] server

[0870] The server is the central location for analyzing the received audio data. It contains the following main software modules:

[0871] Speech recognition engine: Converts received voice data into text data. Google Cloud Speech-to-Text and IBM Watson Speech to Text are available as this engine.

[0872] Natural Language Processing (NLP) module: Analyzes the generated text data and detects specific keywords and phrases related to fraud. This module is powered by IBM Watson Natural Language Understanding.

[0873] Emotion engine: Used to recognize user emotions from recorded voice data. Tools such as Affectiva and Microsoft Azure Emotion Recognition are available.

[0874] Database: Analysis results and audio data are stored. The stored data is used for later review and analysis.

[0875] User

[0876] The users are the elderly themselves and their families. If the server determines that there is a possibility of fraud, the system receives a notification and takes appropriate action.

[0877] Specific examples

[0878] Detection of bank transfer fraud and emotion recognition

[0879] 1. The device detects incoming calls and records the conversation. For example, a scammer might say, "This is my son. I urgently need money, so please transfer it to a specified account right now."

[0880] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[0881] 3. The server passes the voice data to a voice recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer it to the specified account immediately."

[0882] 4. The server analyzes the text data and detects the phrase "Please transfer the money to the specified account." It determines that the data is likely to be fraudulent.

[0883] 5. The server passes the voice data to an emotion engine to recognize the user's emotions (e.g., tension, anxiety). The recognition results are used as factors to further increase the likelihood of fraud.

[0884] 6. The server immediately notifies the user (family member of the elderly person) that "You have received a potentially fraudulent call. The caller says, 'We urgently need money, so please transfer the money to the specified account immediately.' Please be careful as the user's emotions are unstable."

[0885] 7. The user will receive a notification and contact the elderly person to warn them and instruct them not to make the transfer.

[0886] Prepaid Card Fraud Detection and Emotion Recognition

[0887] 1. The device detects an incoming call and records the call. For example, a scammer might ask you to buy a prepaid card at a convenience store and provide the number.

[0888] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[0889] 3. The server passes the voice data to a voice recognition engine, which generates text data such as, "Please purchase a prepaid card at a convenience store and tell me the number."

[0890] 4. The server analyzes the text data and detects the phrase "purchase a prepaid card." It determines that the data is likely fraudulent.

[0891] 5. The server passes the voice data to an emotion engine to recognize the user's emotion (e.g., anxiety, vigilance). The recognition result is used as a factor to further increase the likelihood of fraud.

[0892] 6. The server immediately notifies the user (family member of the elderly person) that "You have received a potentially fraudulent call. The caller asks you to purchase a prepaid card at a convenience store and provide the number. Please be careful as the user's emotions may be unstable."

[0893] 7. The user will receive a notification and contact the elderly person to warn them and provide guidance on how to prevent fraud.

[0894] Through real-time analysis of voice data and emotion recognition, the system is an effective means of protecting the safety of the elderly and can help combat widespread fraud.

[0895] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0896] Step 1:

[0897] When the device detects an incoming call, it automatically starts recording the call. It receives a call start signal as input. Based on this, it converts the voice data during the call into a digital format in real time and temporarily stores the data. Specifically, the recording process starts the moment the phone receiver is lifted, and the display shows "Recording." The output is the recorded digital voice data.

[0898] Step 2:

[0899] When a call ends, the device compresses the recorded voice data. It receives the digital voice data of the ended call as input. Based on this data, it reduces the data size using voice compression techniques such as High Efficiency Audio Codec (HE-AAC). Specifically, the moment the call ends, the message "Compressing voice data..." is displayed. The compressed voice data is temporarily stored on the device. The output is compressed voice data.

[0900] Step 3:

[0901] The device uses a secure protocol (e.g. HTTPS) to send compressed audio data to the server. As input, it receives compressed audio data. Based on this data, it uses encrypted communications to send it to the server while maintaining the privacy and security of the data. Specifically, it displays "Sending data..." and then displays "Transmission complete" once the transmission is complete. The output is the audio data sent to the server.

[0902] Step 4:

[0903] The server passes the received voice data to a voice recognition engine, which converts the voice into text. Compressed voice data is received as input. Based on this, the voice recognition engine (e.g., Google Cloud Speech-to-Text) analyzes the voice data and generates corresponding text data. Specifically, a progress bar appears saying "Analyzing voice data..." as the analysis progresses. The output is the generated text data.

[0904] Step 5:

[0905] The server inputs the generated text data into a natural language processing (NLP) module to detect specific keywords and phrases. As input, it receives text data from a speech recognition engine. This data is analyzed using a natural language processing module (e.g., IBM Watson Natural Language Understanding) to detect keywords related to fraud (e.g., "transfer" and "prepaid card"). Specifically, it displays "Text analysis in progress..." and, if possible fraud is detected, displays "Suspected fraud." The output is a flag or list of the detection results.

[0906] Step 6:

[0907] The server inputs the voice data into an emotion engine to analyze the user's emotions. Recorded voice data is received as input. This data is analyzed by an emotion engine (e.g., Affectiva or Microsoft Azure Emotion Recognition) to recognize the user's emotions (e.g., anxiety, tension, fear) from the tone, pitch, and intensity of the voice. Specifically, the system displays "Analyzing emotions..." and outputs information such as "User emotion: tension." The output is the recognized emotion data.

[0908] Step 7:

[0909] The server integrates the emotion recognition results with the NLP results to comprehensively assess the likelihood of fraud. As input, it receives the detection results of the natural language processing module and the recognition results of the emotion engine. Based on this data, the algorithm evaluates the likelihood of fraud. Specifically, it displays "Evaluating..." and, if the result indicates a high likelihood of fraud, it displays "High likelihood of fraud." The output is a fraud likelihood flag as the evaluation result.

[0910] Step 8:

[0911] The server sends a notification to the elderly person and their family based on the evaluation results. The server receives the fraud possibility evaluation results as input. Based on this data, it generates the notification content and sends it to the elderly person and their family. The specific operation is to display "Sending notification..." and include specific wording of the suspected fraud and a summary of the call. The output is a message indicating that the notification has been sent.

[0912] Step 9:

[0913] The server saves the analysis results and audio data in a database. It receives the evaluation results and audio data as input. Based on this data, it processes the data for storage in the database. Specifically, it displays "Saving data..." and then displays "Saving completed" once the data has been saved. The output is a confirmation message that the data has been saved.

[0914] Step 10:

[0915] The user receives the notification, checks its contents, and contacts the elderly person to take appropriate action. The input is a notification from the server. This notification includes the specific wording of the suspected fraud, a summary of the call, and the results of emotion recognition. The specific action is to check the notification and contact the elderly person to warn them. The output is a message from the user indicating that the action has been taken.

[0916] (Application example 2)

[0917] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0918] Telephone fraud targeting the elderly is increasing year by year, and the methods are becoming more sophisticated. The elderly are particularly vulnerable to methods that create anxiety and fear, making it difficult to prevent fraud damage before it occurs. The purpose of this invention is to provide a system that detects telephone fraud in real time and promptly notifies the elderly and their families, thereby preventing fraud damage.

[0919] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recording voice, means for converting the recorded voice into text, means for analyzing the generated text and detecting specific keywords, means for recognizing emotions from voice data, means for sending notifications based on the detected keywords and emotion recognition results, and means for saving the detection results and voice data. This makes it possible to detect fraud-related keywords and changes in emotions contained in the content of calls in real time and to notify the elderly person and their family members promptly and appropriately.

[0920] A "voice recording device" is a device or component that has the ability to record telephone conversations as digital audio data.

[0921] The "means for converting recorded voice to text" is a voice recognition engine or software that converts recorded voice data into character string data.

[0922] The "means for analyzing the generated text and detecting specific keywords" refers to a natural language processing (NLP) module and analysis algorithms that identify specific terms and phrases within the text data.

[0923] The "means for recognizing emotions from voice data" is an emotion analysis engine that analyzes voice characteristics such as tone, pitch, and intensity to recognize the emotional state of the caller.

[0924] The "means for sending notifications based on detected keywords and emotion recognition results" refers to a communication module or notification system that sends alerts to users and their families in the event of suspected fraud.

[0925] "Means for storing detection results and audio data" refers to a database or storage system that stores the analyzed results and recorded audio data in digital format.

[0926] The present invention relates to a system for monitoring telephone calls in real time and notifying seniors and their families of possible fraud. The system includes a means for recording voice, a means for converting the recorded voice into text, a means for analyzing the generated text and detecting specific keywords, a means for recognizing emotions from the voice data, a means for sending notifications based on the detected keywords and emotion recognition results, and a means for storing the detection results and the voice data.

[0927] A way to record audio

[0928] The telephone terminals used by users are equipped with a function to record calls as digital audio data. This function starts recording from the moment the call starts and saves it when the call ends. A specific example is a built-in application on a smartphone that provides a recording function.

[0929] A means of converting recorded audio into text

[0930] When the call ends, the device sends the recorded voice data to the server, which converts the voice data into text using a speech recognition engine (e.g., Google Speech Recognition API). This saves the call contents as text data.

[0931] A means of analyzing the generated text to detect specific keywords

[0932] The server passes the generated text data to a natural language processing (NLP) module, which is responsible for detecting specific keywords and phrases (e.g., "transfer money" or "prepaid card"). If the NLP module detects keywords that are likely to be fraudulent, the process proceeds to the next step.

[0933] A method for recognizing emotions from voice data

[0934] At the same time, the server analyzes the user's emotions using a sentiment analysis engine (e.g., the sentiment analysis model from the transformers library), which identifies emotions (e.g., anxiety, fear) based on the tone, pitch, and intensity of the voice.

[0935] A means of sending notifications based on detected keywords and emotion recognition results

[0936] The server integrates the detected keywords with the results of emotion recognition, and if there is a high possibility of fraud, it sends a notification to the user and their family. The notification includes the specific content of the call and the detected emotion, and contains information urging the user to take prompt action. Notifications can be sent via email or push notification.

[0937] A means of storing detection results and audio data

[0938] The server stores all analysis results and audio data in a database, which can then be reviewed and shared with law enforcement agencies.

[0939] Specific examples

[0940] For example, if an elderly person receives a call from a scammer saying, "I want to buy a prepaid card at a convenience store and please tell me the number," the following process takes place: The call is recorded and converted into text using speech recognition. Next, the NLP module detects the phrase "I want to buy a prepaid card." At the same time, the sentiment analysis engine recognizes the user's anxiety. As a result, the server determines that there is a high possibility of fraud and sends a notification to the user's family saying, "You have received a potentially fraudulent call. The call says, 'I want to buy a prepaid card at a convenience store and please tell me the number.' Please be careful as the user is in an emotionally unstable state."

[0941] Prompt Sentence Examples

[0942] "Perform the following tasks to generate text from the audio data containing keywords that increase suspicion of fraud and detect the user's emotional state (anxiety, fear, etc.). The transcript of the audio data is as follows: "This is my son. I'd like to buy a prepaid card at a convenience store and would like the number." Based on this, assess the likelihood of fraud and the emotional state."

[0943] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0944] Step 1:

[0945] The device detects an incoming call and activates the call recording function. When the call starts, the device automatically starts recording the conversation as digital audio data. The input is the incoming call signal, and the output is the recorded audio data.

[0946] Step 2:

[0947] When the call ends, the device compresses the recorded voice data and sends it to the server. The device transmits the data using a secure protocol over the Internet, maintaining the privacy and security of the data. The input is the recorded voice data, and the output is a data packet containing the compressed voice data.

[0948] Step 3:

[0949] The server passes the received voice data to a speech recognition engine, which converts the voice into text. The speech recognition engine (for example, Google Speech Recognition API) analyzes the voice data and generates corresponding text data. The input is compressed voice data, and the output is text data.

[0950] Step 4:

[0951] The server inputs the generated text data into a natural language processing (NLP) module, which analyzes the text to detect specific keywords and phrases. The NLP module searches for fraud-related keywords (e.g., "transfer" or "prepaid card") within the text and, if detected, proceeds to the next step. The input is the text data, and the output is the detected keyword information.

[0952] Step 5:

[0953] The server inputs the voice data into an emotion recognition engine to analyze the user's emotional state. The emotion recognition engine (e.g., the emotion analysis model in the transformers library) analyzes voice features such as tone, pitch, and intensity to identify the user's emotion (e.g., anxiety, fear). The input is the voice data, and the output is the emotion recognition result.

[0954] Step 6:

[0955] The server integrates the detected keywords and emotion recognition results to comprehensively assess the likelihood of fraud. Based on this result, if there is a high suspicion of fraud, it prepares to send an alert to the user and their family. The input is the keyword detection information and emotion recognition results, and the output is a notification alert message.

[0956] Step 7:

[0957] The server constructs an alert message and sends it to the specified notification recipients (e.g., the user's family). Notification methods include email and push notifications. The input is the alert message for notification, and the output is the sent notification.

[0958] Step 8:

[0959] The server stores all analysis results and audio recordings in a database, allowing them to be reviewed at a later date and shared with law enforcement if necessary. The inputs are analysis results and audio recordings, and the output is the stored data.

[0960] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0961] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0962] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0963] [Third embodiment]

[0964] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0965] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0966] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0967] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0968] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0969] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0970] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0971] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0972] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0973] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0974] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0975] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0976] ---

[0977] The present invention relates to a system for detecting telephone fraud in real time and notifying elderly people and their families. A specific embodiment of the system and program processing will be described below.

[0978] This system detects the contents of phone calls in real time, analyzes specific keywords, and notifies the elderly and their families. The system consists of the following main components:

[0979] 1. Terminal: A telephone terminal used by the elderly. This terminal records calls and generates audio data.

[0980] 2. Server: Receives voice data, performs speech recognition, generates text data, and analyzes and detects keywords and phrases that may be fraudulent.

[0981] 3. Users: Elderly people and their families. They receive notifications from the server and take appropriate action in case of possible fraud.

[0982] Program processing

[0983] 1. Acquiring audio data

[0984] When the device detects an incoming call, it automatically starts recording the call, and when the call ends, it temporarily saves the recorded audio data.

[0985] 2. Sending audio data

[0986] The device compresses the recorded audio data and sends it to the server using a secure protocol.

[0987] 3. Converting voice data to text

[0988] The server passes the received voice data to a voice recognition engine, converts the voice into text, and obtains the text data generated by the voice recognition engine.

[0989] 4. Natural Language Processing for Fraud Detection

[0990] The server analyzes the text data, detecting specific keywords and phrases (e.g., "transfer" or "prepaid card"), and uses natural language processing techniques to evaluate the content for potential fraud.

[0991] 5. Notification Processing

[0992] Based on the analysis results, the server sends a notification to the elderly person and their family, including the specific wording of the suspected fraudulent call and a summary of the call.

[0993] Users can receive notifications, review the details, and take immediate action, for example, allowing family members to contact the elderly and prevent fraud before it occurs.

[0994] 6. Data Storage and Sharing

[0995] The server stores the detection results and audio data in a database, allowing for detailed analysis and review later if necessary.

[0996] Where necessary, we may share collected data with law enforcement agencies to help them investigate fraud trends and patterns.

[0997] Specific examples

[0998] Detection of bank transfer fraud

[0999] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "This is my son. I urgently need money, so please transfer it to the specified account right now."

[1000] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[1001] 3. The server passes the voice data to a voice recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer it to the specified account immediately."

[1002] 4. The server analyzes the text data and detects the phrase "Transfer the money to the specified account." It determines that the message is likely to be fraudulent.

[1003] 5. The server immediately sends a notification to the user (the elderly person's family member) saying, "You have received a potentially fraudulent call. The message is, 'We urgently need money, so please transfer it to the specified account immediately.' Please be careful."

[1004] 6. The user will receive a notification and contact the elderly person to warn them and instruct them not to make the transfer.

[1005] Prepaid card fraud detection

[1006] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "Please buy a prepaid card at a convenience store and give me the number."

[1007] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[1008] 3. The server passes the voice data to a voice recognition engine, which generates text data such as, "Please purchase a prepaid card at a convenience store and tell me the number."

[1009] 4. The server analyzes the text data and detects the phrase "purchase a prepaid card." It determines that the data is likely fraudulent.

[1010] 5. The server immediately sends a notification to the user (the elderly person's family member) saying, "You have received a potentially fraudulent call. The caller asks you to purchase a prepaid card at a convenience store and provide the number. Please be careful."

[1011] 6. The user will receive a notification and contact the elderly person to warn them and provide guidance on how to prevent fraud.

[1012] Through real-time analysis and notification of voice data, this system is an effective means of protecting the safety of the elderly and is useful in combating widespread fraud.

[1013] ---

[1014] That's all.

[1015] The processing flow will be explained below.

[1016] ---

[1017] Step 1:

[1018] When the device detects an incoming call, it automatically starts recording the call, so that the conversation between the elderly person and the other party is recorded as digital audio data from the moment the call begins.

[1019] ---

[1020] Step 2:

[1021] After the call ends, the device compresses the recorded audio data to improve data transfer efficiency and save storage space. The compressed audio data is then temporarily stored on the device.

[1022] ---

[1023] Step 3:

[1024] The device uses a secure protocol to send compressed audio data to the server, using encrypted communications over the internet to ensure data privacy and security.

[1025] ---

[1026] Step 4:

[1027] The server passes the received voice data to a voice recognition engine, which converts the voice into text. The voice recognition engine then analyzes the voice data and outputs it as "character string data."

[1028] ---

[1029] Step 5:

[1030] The server then inputs the generated text data into a natural language processing (NLP) module, which analyzes the text and detects specific keywords and phrases, such as whether it contains terms associated with fraud, such as "transfer" or "prepaid card."

[1031] ---

[1032] Step 6:

[1033] The server evaluates the analysis results from the NLP module and, if it determines that there is a high possibility of fraud, begins preparing a notification. At this stage, the notification content is generated and a message is created to be sent to the elderly person and their family.

[1034] ---

[1035] Step 7:

[1036] The server sends a notification message, which can be sent to the user via SMS, email, or a dedicated app, allowing the user to quickly become aware of potential fraud and take action.

[1037] ---

[1038] Step 8:

[1039] The user receives a notification, which includes the suspected fraudulent text and a summary of the call, and can then contact the senior to provide guidance or advice.

[1040] ---

[1041] Step 9:

[1042] The server stores the analysis results and audio data in a database, which can be used for later review and analysis.

[1043] ---

[1044] Step 10:

[1045] If necessary, the server will initiate the process of sharing collected data with law enforcement agencies to investigate fraud trends and patterns and assist in legal action.

[1046] ---

[1047] That's all.

[1048] Example 1

[1049] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1050] In recent years, telephone fraud has been on the rise, with elderly people being particularly vulnerable. To prevent fraud, it is necessary to detect suspicious calls in real time and promptly notify the elderly person and their family. However, conventional systems are unable to adequately perform this real-time detection and notification, which often results in damage occurring. For this reason, there is a need for a system that can detect and notify fraud more effectively and quickly.

[1051] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1052] In this invention, the server includes a means for compressing voice data and transmitting it using a secure protocol, a means for converting recorded voice into text, a natural language processing means for analyzing the generated text and detecting specific keywords, a means for sending a notification based on the detected keywords, a means for sending the notification to the user, a means for saving the analysis results and the voice data, and a means for sharing the analysis results and the voice data with law enforcement agencies as needed. This makes it possible to detect suspected fraudulent calls in real time and quickly notify the elderly person and their family.

[1053] "Audio recording means" refers to a device or program that automatically records telephone conversations and saves the audio data.

[1054] "Means for compressing recorded audio data and transmitting it using a secure protocol" refers to a device or program that compresses recorded audio files using a compression algorithm and transmits them to a server using a secure communication protocol such as HTTPS or SSL.

[1055] A "means for converting recorded speech to text" is a system or program that uses a speech recognition engine to analyze speech data and convert it into a human-understandable text format.

[1056] "Natural language processing means for analyzing generated text and detecting specific keywords" means a system or program that uses natural language processing techniques to analyze text data and detect specific keywords or phrases that may be suspected of being fraudulent.

[1057] "Means for sending notifications based on detected keywords" means a system or program for sending warnings or notifications to users when detected keywords or phrases indicate suspected fraud.

[1058] "Means for sending notifications to users" means a system or program for sending notifications to the elderly person and their family members via SMS, email, or other communication means.

[1059] "Means for storing analysis results and audio data" refers to a system or program for storing analysis results and audio data in a database or file system, and keeping them in a state where they can be accessed and analyzed later.

[1060] "Means for sharing analytics results and audio data with law enforcement as needed" means a system or program that shares stored analytics results and audio data with law enforcement to assist in fraud investigation and prevention.

[1061] "Communication terminals used by the elderly" refer to telephones, smartphones, or other communication devices capable of making calls that are primarily used by the elderly in their daily lives.

[1062] A "server with a speech recognition engine" is a server that has speech recognition capabilities to convert received speech data into text, and is a device that has the computing resources to execute programs and algorithms to analyze speech.

[1063] The present invention relates to a system for detecting telephone fraud in real time and notifying elderly people and their families of the fraud. Specific embodiments of the system will be described below.

[1064] This system consists of the following main hardware and software:

[1065] Hardware and Software Configuration

[1066] 1. Device:

[1067] Telephones and smartphones are used as communication devices by the elderly. These devices record calls and generate audio data.

[1068] The software used is a module for recording (for example, a recording class developed in Java or Swift).

[1069] 2. Server:

[1070] A server that receives voice data, performs speech recognition, and generates text data. The server is equipped with a speech recognition engine (e.g., Google Speech-to-Text API) and a natural language processing engine (e.g., SpaCy or NLTK).

[1071] To run the program, web frameworks such as Python, Flask, and Django are used.

[1072] 3. User:

[1073] The elderly person and their family members receive notifications from the server and take appropriate action if there is a possibility of fraud.

[1074] Data processing and calculation

[1075] Acquiring audio data

[1076] When the device detects an incoming call, it automatically starts recording the call. For example, you can use the smartphone's built-in app or a dedicated recording app. When the call ends, the recorded audio data is temporarily stored on the device.

[1077] Sending audio data

[1078] The device compresses the recorded audio data and sends it to the server using a secure protocol (e.g., HTTPS or SSL) using an algorithm such as GZIP.

[1079] Converting audio data to text

[1080] The server passes the received voice data to a speech recognition engine (e.g., Google Speech-to-Text API) and converts the voice into text, which is then stored in a database.

[1081] Natural Language Processing for Fraud Detection

[1082] The server analyzes the text data using a natural language processing engine (e.g., SpaCy or NLTK) to detect specific keywords and phrases (e.g., "transfer," "prepaid card," etc.).

[1083] Notification Processing

[1084] Based on the analysis results, the server sends a notification to the elderly person and their family, including specific wording of the suspected fraud and a summary of the call. For example, notifications are sent via SMS or email using Twilio or SendGrid.

[1085] Data storage and sharing

[1086] The server stores the detection results and audio data in a database using MySQL or PostgreSQL. The stored data can be analyzed and reviewed in detail at a later date, and can be shared with law enforcement agencies if necessary.

[1087] Specific examples

[1088] Detection of bank transfer fraud

[1089] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "This is my son. I urgently need money, so please transfer it to the specified account right now."

[1090] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[1091] 3. The server passes the voice data to a voice recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer it to the specified account immediately."

[1092] 4. The server analyzes the text data and detects the phrase "Transfer the money to the specified account." It determines that the message is likely to be fraudulent.

[1093] 5. The server immediately sends a notification to the user (the elderly person's family member) saying, "You have received a potentially fraudulent call. The message is, 'We urgently need money, so please transfer it to the specified account immediately.' Please be careful."

[1094] 6. The user receives a notification and contacts the elderly person to warn them and prevent any damage from occurring.

[1095] Prepaid card fraud detection

[1096] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "Please buy a prepaid card at a convenience store and give me the number."

[1097] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[1098] 3. The server passes the voice data to a voice recognition engine, which generates text data such as, "Please purchase a prepaid card at a convenience store and tell me the number."

[1099] 4. The server analyzes the text data and detects the phrase "purchase a prepaid card." It determines that the data is likely fraudulent.

[1100] 5. The server immediately sends a notification to the user (the elderly person's family member) saying, "You have received a potentially fraudulent call. The caller asks you to purchase a prepaid card at a convenience store and provide the number. Please be careful."

[1101] 6. The user receives a notification and contacts the elderly person to warn them and prevent any damage from occurring.

[1102] Prompt Sentence Examples

[1103] In the case of detecting bank transfer fraud

[1104] "Please tell me how to build a system that notifies the elderly person and their family when they receive a call from someone pretending to be their son requesting that they transfer money to a specified account."

[1105] Prepaid card fraud detection

[1106] "Please tell me how to build a system that notifies elderly people and their families when they purchase a prepaid card at a convenience store and are asked to provide the card number."

[1107] The system provides real-time voice analysis and rapid notification to protect against fraud targeting seniors, making it effective in combating a wide range of frauds.

[1108] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1109] Step 1: Acquire audio data

[1110] When the device detects an incoming call, it automatically starts recording the call, recording the contents of the call in real time.

[1111] Input: Call audio

[1112] Data processing: Save call audio as a recording file

[1113] Output: Recording file

[1114] How it works: The device's calling app triggers an incoming call event, calls a recording module (for example, a recording class written in Java or Swift) to start recording, and then saves the recording file in the device's experience data storage after the call ends.

[1115] Step 2: Sending audio data

[1116] The device compresses the recorded audio data and sends it to the server using a secure protocol.

[1117] Input: Recording file

[1118] Data processing: Compress the audio data using GZIP or other methods and package it for HTTPS requests

[1119] Output: Compressed audio data is sent to the server

[1120] What it does: It compresses the recorded audio data using the GZIP compression algorithm and sends it to the server's API endpoint using the HTTPS protocol, for example, using Python's requests library or Java's HttpURLConnection class.

[1121] Step 3: Convert audio data to text

[1122] The server passes the received voice data to a voice recognition engine, which converts the voice into text.

[1123] Input: Compressed audio data

[1124] Data processing: Converting speech to text using a speech recognition engine (such as Google Speech-to-Text API)

[1125] Output: Text data

[1126] What happens: The server receives the audio data, decompresses it, and passes it to the speech recognition engine. The speech recognition engine analyzes the audio file and generates text data, which is then stored in a database.

[1127] Step 4: Natural Language Processing for Fraud Detection

[1128] The server analyzes the text data using a natural language processing engine to detect specific keywords and phrases.

[1129] Input: Text data

[1130] Data processing: Use natural language processing engines (such as SpaCy or NLTK) to extract and analyze specific keywords and phrases within text data.

[1131] Output: Analysis results showing suspected fraudulent phrases and keywords

[1132] How it works: The server passes the text data to a natural language processing engine, which analyzes it for potentially fraudulent keywords and phrases. Based on the analysis, any data that is suspected to be fraudulent is flagged and stored in a database.

[1133] Step 5: Notification Processing

[1134] Based on the analysis results, the server sends a notification to the elderly person and their family.

[1135] Input: Analysis results

[1136] Data processing: Generate notification content and send it using a notification service such as Twilio or SendGrid

[1137] Output: Email and SMS notifications sent to the elderly person and their family

[1138] Specific operation: The server composes the analysis results in JSON format and sends emails or SMS to the elderly person and their family members through existing notification services such as Twilio and SendGrid.

[1139] Step 6: Data storage and sharing

[1140] The server stores the detection results and audio data in a database, which may be shared with law enforcement agencies if necessary.

[1141] Input: Analysis results and audio data

[1142] Data processing: Analysis results and audio data are written to a database and shared with law enforcement agencies as needed

[1143] Output: Analysis results and audio data stored in a database

[1144] What it does: The server uses an ORM (Object-Relational Mapping) tool such as SQLAlchemy to write the analysis results and audio data to a database (such as MySQL or PostgreSQL), and provides access and data sharing to law enforcement agencies as needed.

[1145] These steps will allow the entire system to work together to protect seniors from phone scams.

[1146] (Application example 1)

[1147] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1148] In recent years, frauds targeting the elderly have been increasing. However, many elderly people do not have sufficient means to avoid fraud, and the damage caused by these frauds is becoming more serious. To solve this problem, a system is needed that can detect telephone frauds in real time and quickly notify the elderly and their families.

[1149] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1150] In this invention, the server includes a means for transmitting recorded voice data to a remote server using a security protocol, a means for converting the voice data into text using a voice recognition engine, a means for analyzing the text data using natural language processing technology, and a means for sending a push notification to a smartphone, thereby enabling real-time detection of suspected fraud and prompt notification to the elderly person and their family.

[1151] "Audio recording means" means a device or function that records the contents of a telephone conversation as digital audio data.

[1152] "Means for converting recorded voice into text" refers to a technology for converting voice data into text in a natural language, i.e., a device or function that generates text data from voice using a voice recognition engine.

[1153] "Means for analyzing the generated text and detecting specific keywords" refers to a device or function that analyzes the generated text data using algorithms or natural language processing techniques to identify keywords or phrases that may be fraudulent.

[1154] A "means for sending notifications based on detected keywords" is a device or function that, when a keyword that indicates suspected fraud is detected, notifies the user or their family of the potential fraud via a notification mechanism.

[1155] "Means for storing detection results and audio data" refers to a device or function that stores the analyzed text data and original audio data in a database or storage system so that they can be accessed later.

[1156] "Means for transmitting recorded audio data to a remote server using a secure protocol" refers to a device or function that compresses the recorded audio data and transfers it to a remote server using a secure communication protocol (e.g., HTTPS).

[1157] "Means for converting voice data into text using a voice recognition engine" refers to a device or function that mechanically converts voice data into text using voice recognition software or a cloud service.

[1158] A "means for analyzing text data using natural language processing techniques" is a device or function that uses natural language processing algorithms or models to evaluate text data and identify keywords or phrases that may contain indicators of fraud.

[1159] A "means for sending push notifications to a smartphone" is a device or system with push notification functionality for quickly notifying users and their families when potential fraud is detected.

[1160] The present invention relates to a system for detecting telephone fraud in real time and notifying elderly people and their families promptly. The following describes a specific embodiment and the hardware and software used.

[1161] System configuration

[1162] The system consists of the following main components:

[1163] 1. Device:

[1164] This is a telephone terminal used by elderly people. It records calls and generates audio data.

[1165] This includes communication devices such as smartphones with call recording functionality installed.

[1166] 2. Server:

[1167] Voice data is received, voice recognition is performed, and text data is generated.

[1168] Specific keywords are analyzed using natural language processing technology.

[1169] Google Cloud Speech-to-Text and Amazon Transcribe are used as speech recognition engines.

[1170] Natural language processing technologies used include Google Cloud Natural Language API and Amazon Comprehend.

[1171] 3. User:

[1172] The elderly themselves and their families.

[1173] Receive notifications from the server and take appropriate action in case of potential fraud.

[1174] About program processing

[1175] The processing flow by the system is shown below.

[1176] 1. Acquire audio data:

[1177] When the device detects an incoming call, it automatically records the call. When the call ends, it stops recording and temporarily saves the audio data.

[1178] 2. Sending audio data:

[1179] The device compresses the recorded audio data and sends it to the server using a secure communication protocol (e.g., HTTPS).

[1180] 3. Audio to text conversion:

[1181] The server passes the received voice data to a speech recognition engine, converts the voice into text data, and generates highly accurate text data from the voice data using Google Cloud Speech-to-Text or Amazon Transcribe.

[1182] 4. Natural Language Processing for Fraud Detection:

[1183] The server analyzes the text data using natural language processing technology (Google Cloud Natural Language API and Amazon Comprehend) to detect keywords and phrases that may be fraudulent.

[1184] 5. Notification Processing:

[1185] Based on the analysis results, the server sends a push notification to the smartphone, and also notifies the user and their family members via email and SMS.

[1186] Use Firebase Cloud Messaging or Amazon SNS for push notification.

[1187] 6. Data Storage and Sharing:

[1188] The server stores the analysis results and audio data in a cloud database (Firebase Firestore or Amazon DynamoDB), which allows the information to be shared with law enforcement agencies if necessary.

[1189] Specific examples

[1190] Example of detecting bank transfer fraud

[1191] The device detects an incoming call and records the conversation. The scammer says, "This is my son. I urgently need money, so please transfer the money to my specified account now." When the call ends, the recorded audio data is compressed and sent to the server. The server passes the audio data to a speech recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer the money to my specified account now." The server analyzes the text data and detects the phrase "transfer the money to my specified account." It immediately sends a notification to the user saying, "You have received a potentially fraudulent call. The message is, 'I urgently need money, so please transfer the money to my specified account now.' Please be careful."

[1192] Prepaid card fraud detection example

[1193] The device detects an incoming call and records the conversation. The scammer says, "Please buy a prepaid card at a convenience store and give me the number." When the call ends, the recorded audio data is compressed and sent to the server. The server passes the audio data to a speech recognition engine, which generates text data saying, "Please buy a prepaid card at a convenience store and give me the number." The server analyzes the text data and detects the phrase "please buy a prepaid card." It immediately sends a notification to the user saying, "You have received a potentially fraudulent call. The message is, 'Please buy a prepaid card at a convenience store and give me the number.' Please be careful."

[1194] In this way, this system can protect the elderly from fraud through real-time analysis and notification of voice data.

[1195] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1196] Step 1:

[1197] The device detects an incoming call and automatically starts recording the call. When the call ends, the recording stops and the audio data is temporarily saved.

[1198] Specifically, the call recording function installed on the smartphone is used to save the audio data during the call in a digital format, such as a .wav or .3gp file.

[1199] Step 2:

[1200] The device compresses the recorded audio data and sends it to the server using a secure communication protocol (e.g., HTTPS). The audio data is compressed to transmit the audio file efficiently using less bandwidth.

[1201] Input: Recorded audio data

[1202] Output: Compressed audio data file

[1203] Specifically, the recorded audio data is converted into a compressed format such as .mp3 or .opus and uploaded to a server using the HTTPS protocol.

[1204] Step 3:

[1205] The server passes the received voice data to a voice recognition engine, which converts the voice into text.

[1206] Input: Compressed audio data file

[1207] Output: Text data

[1208] Specifically, the voice data is converted into text data using the speech recognition engines of Google Cloud Speech-to-Text and Amazon Transcribe. The server calls the API to send the audio file and obtains the returned text data.

[1209] Step 4:

[1210] The server analyzes the text data using natural language processing technology to detect specific keywords and phrases.

[1211] Input: Text data

[1212] Output: Analysis results containing keywords and phrases that may be fraudulent

[1213] Specifically, the system analyzes text data using Google Cloud Natural Language API and Amazon Comprehend to detect fraudulent keywords such as "transfer" and "prepaid card."

[1214] Step 5:

[1215] Based on the analysis results, the server sends a push notification to the smartphone, and also notifies the user and their family via email or SMS.

[1216] Input: Suspected fraud analysis results

[1217] Output: Push notification and email / SMS notification

[1218] Specifically, it uses Firebase Cloud Messaging and Amazon SNS to send push notifications to smartphones, as well as email and SMS notifications to users and their families, containing specific details about the suspected scam and a warning message.

[1219] Step 6:

[1220] The server stores the detection results and audio data in a cloud database (such as Firebase Firestore or Amazon DynamoDB) and shares the data with law enforcement agencies if necessary.

[1221] Input: Analysis results and audio data

[1222] Output: Data stored in a cloud database

[1223] Specifically, the system securely transmits and stores voice data and analysis results in a cloud database, and provides authenticated access for data sharing with law enforcement agencies.

[1224] In this way, the individual processing steps work together to effectively operate a system that protects the elderly from fraud.

[1225] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1226] ---

[1227] This invention relates to a system that detects telephone fraud in real time and notifies the elderly person and their family. In particular, by combining it with an emotion engine that recognizes the user's emotions from their voice, the possibility of fraud can be further increased. This system is implemented with the following components.

[1228] This system detects the content of phone conversations in real time, analyzes specific keywords, recognizes the user's emotions, and notifies the elderly and their families.The system consists of the following main components.

[1229] 1. Terminal: A telephone terminal used by the elderly. This terminal records calls and generates audio data.

[1230] 2. Server: Receives voice data, performs speech recognition, generates text data, analyzes keywords and phrases that may be fraudulent, and recognizes user emotions.

[1231] 3. Emotion Engine: Recognizes user emotions from recorded voice data. Emotion recognition analyzes the emotional tone of a call to assess the likelihood of fraud.

[1232] 4. Users: Elderly people and their families. They receive notifications from the server and take appropriate action in case of possible fraud.

[1233] Program processing

[1234] 1. Acquiring audio data

[1235] When the device detects an incoming call, it automatically starts recording the call, so that the conversation between the elderly person and the other party is recorded as digital audio data from the moment the call begins.

[1236] 2. Sending audio data

[1237] After the call ends, the device compresses the recorded audio data to improve data transfer efficiency and save storage space. The compressed audio data is then temporarily stored on the device.

[1238] The device uses a secure protocol to send compressed audio data to the server, using encrypted communications over the internet to ensure data privacy and security.

[1239] 3. Converting voice data to text

[1240] The server passes the received voice data to a voice recognition engine, which converts the voice into text. The voice recognition engine then analyzes the voice data and outputs it as "character string data."

[1241] 4. Natural Language Processing for Fraud Detection

[1242] The server then inputs the generated text data into a natural language processing (NLP) module, which analyzes the text and detects specific keywords and phrases, such as whether it contains terms associated with fraud, such as "transfer" or "prepaid card."

[1243] 5. Emotion recognition

[1244] The server inputs the voice data into the emotion engine and analyzes the user's emotions. The emotion engine recognizes the user's emotions (e.g., anxiety, tension, fear) from the tone, pitch, and intensity of the voice.

[1245] The server integrates the emotion recognition results with the NLP results to provide a comprehensive assessment of the likelihood of fraud.

[1246] 6. Notification Processing

[1247] Based on the analysis results, the server sends a notification to the elderly person and their family, including the specific wording of the suspected fraudulent call and a summary of the call.

[1248] The server then adjusts the notification content based on the results of emotion recognition. For example, if the user's emotions are very unstable, the notification will include an alert urging immediate action.

[1249] 7. User Response

[1250] The user receives a notification and checks its contents, which include the suspected fraudulent text, a summary of the call, and the results of emotion recognition. The user can then contact the elderly person and take action if necessary.

[1251] 8. Data Storage and Sharing

[1252] The server stores the analysis results and audio data in a database, which can be used for later review and analysis.

[1253] Where necessary, we may share collected data with law enforcement agencies to help them investigate fraudulent trends and patterns.

[1254] Specific examples

[1255] Detection of bank transfer fraud and emotion recognition

[1256] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "This is my son. I urgently need money, so please transfer it to the specified account right now."

[1257] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[1258] 3. The server passes the voice data to a voice recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer it to the specified account immediately."

[1259] 4. The server analyzes the text data and detects the phrase "Transfer the money to the specified account." It determines that the message is likely to be fraudulent.

[1260] 5. The server passes the voice data to an emotion engine to recognize the user's emotions (e.g., nervousness, anxiety). The recognition results are evaluated as factors that further increase the likelihood of fraud.

[1261] 6. The server immediately sends a notification to the user (family member of the elderly person) saying, "You have received a potentially fraudulent call. The message is, 'We urgently need money, so please transfer it to the specified account immediately.' Please be careful as the user's emotions are unstable."

[1262] 7. The user will receive a notification and contact the elderly person to warn them and instruct them not to make the transfer.

[1263] Prepaid Card Fraud Detection and Emotion Recognition

[1264] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "Please buy a prepaid card at a convenience store and give me the number."

[1265] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[1266] 3. The server passes the voice data to a voice recognition engine, which generates text data such as, "Please purchase a prepaid card at a convenience store and tell me the number."

[1267] 4. The server analyzes the text data and detects the phrase "purchase a prepaid card." It determines that the data is likely fraudulent.

[1268] 5. The server passes the voice data to an emotion engine to recognize the user's emotion (e.g., anxiety, alarm). The recognition result is evaluated as a factor that further increases the likelihood of fraud.

[1269] 6. The server immediately sends a notification to the user (family member of the elderly person) saying, "You have received a potentially fraudulent call. The caller asks you to purchase a prepaid card at a convenience store and provide the number. Please be careful as the user's emotions are unstable."

[1270] 7. The user will receive a notification and contact the elderly person to warn them and provide guidance on how to prevent fraud.

[1271] Through real-time analysis of voice data and emotion recognition, this system is an effective means of protecting the safety of the elderly and can be useful in combating widespread fraud.

[1272] ---

[1273] That's all.

[1274] The processing flow will be explained below.

[1275] ---

[1276] Step 1:

[1277] When the device detects an incoming call, it automatically starts recording the call and continues recording until the call ends, saving the conversation as digital audio data.

[1278] ---

[1279] Step 2:

[1280] After the call ends, the device compresses the recorded audio data, which saves storage space and improves data transmission efficiency.

[1281] ---

[1282] Step 3:

[1283] The device uses a secure protocol to send compressed audio data to the server, where it is encrypted and securely transmitted over the internet.

[1284] ---

[1285] Step 4:

[1286] The server passes the received voice data to a speech recognition engine, which converts the voice into text. The speech recognition engine analyzes the voice data and generates corresponding text data.

[1287] ---

[1288] Step 5:

[1289] The server inputs the generated text data into a natural language processing (NLP) module, which analyzes the text data and detects specific keywords and phrases related to fraud (e.g., "transfer money" or "prepaid card").

[1290] ---

[1291] Step 6:

[1292] The server passes the voice data to the emotion engine to recognize the user's emotion. The emotion engine analyzes the voice tone, pitch, intensity, etc. to identify the user's emotion (e.g., anxiety, tension, fear, etc.).

[1293] ---

[1294] Step 7:

[1295] The server combines the analysis results of the NLP module and the recognition results of the emotion engine to comprehensively evaluate the likelihood of fraud. If the emotion engine's results increase suspicion of fraud, the evaluation is revised upward.

[1296] ---

[1297] Step 8:

[1298] If the server determines that fraud is likely, it prepares a notification message containing the suspected fraudulent text detected, a summary of the call, and the results of emotion recognition.

[1299] ---

[1300] Step 9:

[1301] The server sends a notification message to the user (elderly person and their family) via SMS, email, or a dedicated app.

[1302] ---

[1303] Step 10:

[1304] The user receives and reviews the notification, which includes the specific suspected fraudulent text, a summary of the call, and the results of emotion recognition. The user then contacts the senior and takes appropriate action to address the potential fraud.

[1305] ---

[1306] Step 11:

[1307] The server stores the analysis results and audio data in a database, making it easy to check and analyze later.

[1308] ---

[1309] Step 12:

[1310] If necessary, the server will initiate the process of sharing collected data with law enforcement agencies to investigate fraudulent trends and patterns and assist in legal action.

[1311] ---

[1312] That's all.

[1313] Example 2

[1314] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1315] Conventional telephone fraud prevention systems are limited in their ability to record and analyze call content, making it difficult to detect fraud in real time. Furthermore, they are unable to take into account the emotions of the elderly, such as anxiety and tension, which can lead to reduced accuracy in detecting fraudulent activity. Furthermore, notification functions are inadequate, leaving a lack of means to quickly alert the elderly and their families. There is a need for a new system that can solve these problems.

[1316] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for recording voice, a means for compressing and transmitting the recorded voice, a means for converting the transmitted voice into text using a voice recognition engine, a means for analyzing the generated text using a natural language processing module and detecting specific keywords, a means for recognizing the user's emotions from the voice data, a means for comprehensively evaluating the possibility of fraud, a means for sending a notification based on the evaluation result, and a means for saving the detection result and the voice data. This makes it possible to analyze the content of the call and the user's emotions in real time, identify the possibility of fraud with high accuracy, and promptly notify the elderly person and their family.

[1317] A "voice recording means" is a device capable of recording voice data in digital format at the initiation of a telephone call.

[1318] The "means for compressing and transmitting recorded audio" is a device that has the function of compressing recorded audio data in a manner that reduces the data volume and transmitting it to a server using a secure protocol.

[1319] The "means for converting transmitted voice into text using a voice recognition engine" refers to software or hardware for analyzing received voice data on a server and converting it into corresponding text data.

[1320] "Means for analyzing the generated text using a natural language processing module and detecting specific keywords" refers to a software module that has the function of analyzing text data using natural language processing technology and detecting specific keywords and phrases related to fraud.

[1321] "Means for recognizing user emotions from voice data" refers to a system or software that analyzes the tone, pitch, intensity, etc. of recorded voice data to recognize the user's emotional state (e.g., anxiety, tension, fear).

[1322] "Means for comprehensively assessing the likelihood of fraud" refers to algorithms or systems that integrate the results of keyword detection using natural language processing and emotion recognition to determine the likelihood of fraud.

[1323] The "means for sending notifications based on the evaluation results" refers to a system that has the function of sending notifications containing warnings and detailed information to the elderly person and their family if it is determined that there is a high possibility of fraud.

[1324] The "means for storing the detection results and audio data" refers to a device or system that has the function of storing the analysis results and recorded audio data in storage such as a database for later confirmation or analysis.

[1325] This invention relates to a system that detects telephone fraud in real time and notifies the elderly person and their family. The main purpose of this system is to record voice, analyze the recorded voice, and evaluate the possibility of fraud with high accuracy. In particular, by incorporating a mechanism to recognize the user's emotions from voice data, the accuracy of fraud detection can be further improved. This system is realized using the following specific hardware and software.

[1326] Hardware and Software Configuration

[1327] Terminal

[1328] The telephone terminals used by the elderly have the ability to record phone conversations in real time. The terminals record conversations as digital audio data from the moment the call begins until it ends. The recorded audio data is compressed for efficient storage and sent to a server using a secure protocol (e.g., HTTPS).

[1329] server

[1330] The server is the central location for analyzing the received audio data. It contains the following main software modules:

[1331] Speech recognition engine: Converts received voice data into text data. Google Cloud Speech-to-Text and IBM Watson Speech to Text are available as this engine.

[1332] Natural Language Processing (NLP) module: Analyzes the generated text data and detects specific keywords and phrases related to fraud. This module is powered by IBM Watson Natural Language Understanding.

[1333] Emotion engine: Used to recognize user emotions from recorded voice data. Tools such as Affectiva and Microsoft Azure Emotion Recognition are available.

[1334] Database: Analysis results and audio data are stored. The stored data is used for later review and analysis.

[1335] User

[1336] The users are the elderly themselves and their families. If the server determines that there is a possibility of fraud, the system receives a notification and takes appropriate action.

[1337] Specific examples

[1338] Detection of bank transfer fraud and emotion recognition

[1339] 1. The device detects incoming calls and records the conversation. For example, a scammer might say, "This is my son. I urgently need money, so please transfer it to a specified account right now."

[1340] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[1341] 3. The server passes the voice data to a voice recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer it to the specified account immediately."

[1342] 4. The server analyzes the text data and detects the phrase "Please transfer the money to the specified account." It determines that the data is likely to be fraudulent.

[1343] 5. The server passes the voice data to an emotion engine to recognize the user's emotions (e.g., tension, anxiety). The recognition results are used as factors to further increase the likelihood of fraud.

[1344] 6. The server immediately notifies the user (family member of the elderly person) that "You have received a potentially fraudulent call. The caller says, 'We urgently need money, so please transfer the money to the specified account immediately.' Please be careful as the user's emotions are unstable."

[1345] 7. The user will receive a notification and contact the elderly person to warn them and instruct them not to make the transfer.

[1346] Prepaid Card Fraud Detection and Emotion Recognition

[1347] 1. The device detects an incoming call and records the call. For example, a scammer might ask you to buy a prepaid card at a convenience store and provide the number.

[1348] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[1349] 3. The server passes the voice data to a voice recognition engine, which generates text data such as, "Please purchase a prepaid card at a convenience store and tell me the number."

[1350] 4. The server analyzes the text data and detects the phrase "purchase a prepaid card." It determines that the data is likely fraudulent.

[1351] 5. The server passes the voice data to an emotion engine to recognize the user's emotion (e.g., anxiety, vigilance). The recognition result is used as a factor to further increase the likelihood of fraud.

[1352] 6. The server immediately notifies the user (family member of the elderly person) that "You have received a potentially fraudulent call. The caller asks you to purchase a prepaid card at a convenience store and provide the number. Please be careful as the user's emotions may be unstable."

[1353] 7. The user will receive a notification and contact the elderly person to warn them and provide guidance on how to prevent fraud.

[1354] Through real-time analysis of voice data and emotion recognition, the system is an effective means of protecting the safety of the elderly and can help combat widespread fraud.

[1355] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1356] Step 1:

[1357] When the device detects an incoming call, it automatically starts recording the call. It receives a call start signal as input. Based on this, it converts the voice data during the call into a digital format in real time and temporarily stores the data. Specifically, the recording process starts the moment the phone receiver is lifted, and the display shows "Recording." The output is the recorded digital voice data.

[1358] Step 2:

[1359] When a call ends, the device compresses the recorded voice data. It receives the digital voice data of the ended call as input. Based on this data, it reduces the data size using voice compression techniques such as High Efficiency Audio Codec (HE-AAC). Specifically, the moment the call ends, the message "Compressing voice data..." is displayed. The compressed voice data is temporarily stored on the device. The output is compressed voice data.

[1360] Step 3:

[1361] The device uses a secure protocol (e.g. HTTPS) to send compressed audio data to the server. As input, it receives compressed audio data. Based on this data, it uses encrypted communications to send it to the server while maintaining the privacy and security of the data. Specifically, it displays "Sending data..." and then displays "Transmission complete" once the transmission is complete. The output is the audio data sent to the server.

[1362] Step 4:

[1363] The server passes the received voice data to a voice recognition engine, which converts the voice into text. Compressed voice data is received as input. Based on this, the voice recognition engine (e.g., Google Cloud Speech-to-Text) analyzes the voice data and generates corresponding text data. Specifically, a progress bar appears saying "Analyzing voice data..." as the analysis progresses. The output is the generated text data.

[1364] Step 5:

[1365] The server inputs the generated text data into a natural language processing (NLP) module to detect specific keywords and phrases. As input, it receives text data from a speech recognition engine. This data is analyzed using a natural language processing module (e.g., IBM Watson Natural Language Understanding) to detect keywords related to fraud (e.g., "transfer" and "prepaid card"). Specifically, it displays "Text analysis in progress..." and, if possible fraud is detected, displays "Suspected fraud." The output is a flag or list of the detection results.

[1366] Step 6:

[1367] The server inputs the voice data into an emotion engine to analyze the user's emotions. Recorded voice data is received as input. This data is analyzed by an emotion engine (e.g., Affectiva or Microsoft Azure Emotion Recognition) to recognize the user's emotions (e.g., anxiety, tension, fear) from the tone, pitch, and intensity of the voice. Specifically, the system displays "Analyzing emotions..." and outputs information such as "User emotion: tension." The output is the recognized emotion data.

[1368] Step 7:

[1369] The server integrates the emotion recognition results with the NLP results to comprehensively assess the likelihood of fraud. As input, it receives the detection results of the natural language processing module and the recognition results of the emotion engine. Based on this data, the algorithm evaluates the likelihood of fraud. Specifically, it displays "Evaluating..." and, if the result indicates a high likelihood of fraud, it displays "High likelihood of fraud." The output is a fraud likelihood flag as the evaluation result.

[1370] Step 8:

[1371] The server sends a notification to the elderly person and their family based on the evaluation results. The server receives the fraud possibility evaluation results as input. Based on this data, it generates the notification content and sends it to the elderly person and their family. The specific operation is to display "Sending notification..." and include specific wording of the suspected fraud and a summary of the call. The output is a message indicating that the notification has been sent.

[1372] Step 9:

[1373] The server saves the analysis results and audio data in a database. It receives the evaluation results and audio data as input. Based on this data, it processes the data for storage in the database. Specifically, it displays "Saving data..." and then displays "Saving completed" once the data has been saved. The output is a confirmation message that the data has been saved.

[1374] Step 10:

[1375] The user receives the notification, checks its contents, and contacts the elderly person to take appropriate action. The input is a notification from the server. This notification includes the specific wording of the suspected fraud, a summary of the call, and the results of emotion recognition. The specific action is to check the notification and contact the elderly person to warn them. The output is a message from the user indicating that the action has been taken.

[1376] (Application example 2)

[1377] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1378] Telephone fraud targeting the elderly is increasing year by year, and the methods are becoming more sophisticated. The elderly are particularly vulnerable to methods that create anxiety and fear, making it difficult to prevent fraud damage before it occurs. The purpose of this invention is to provide a system that detects telephone fraud in real time and promptly notifies the elderly and their families, thereby preventing fraud damage.

[1379] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recording voice, means for converting the recorded voice into text, means for analyzing the generated text and detecting specific keywords, means for recognizing emotions from voice data, means for sending notifications based on the detected keywords and emotion recognition results, and means for saving the detection results and voice data. This makes it possible to detect fraud-related keywords and changes in emotions contained in the content of calls in real time and to notify the elderly person and their family members promptly and appropriately.

[1380] A "voice recording device" is a device or component that has the ability to record telephone conversations as digital audio data.

[1381] The "means for converting recorded voice to text" is a voice recognition engine or software that converts recorded voice data into character string data.

[1382] The "means for analyzing the generated text and detecting specific keywords" refers to a natural language processing (NLP) module and analysis algorithms that identify specific terms and phrases within the text data.

[1383] The "means for recognizing emotions from voice data" is an emotion analysis engine that analyzes voice characteristics such as tone, pitch, and intensity to recognize the emotional state of the caller.

[1384] The "means for sending notifications based on detected keywords and emotion recognition results" refers to a communication module or notification system that sends alerts to users and their families in the event of suspected fraud.

[1385] "Means for storing detection results and audio data" refers to a database or storage system that stores the analyzed results and recorded audio data in digital format.

[1386] The present invention relates to a system for monitoring telephone calls in real time and notifying seniors and their families of possible fraud. The system includes a means for recording voice, a means for converting the recorded voice into text, a means for analyzing the generated text and detecting specific keywords, a means for recognizing emotions from the voice data, a means for sending notifications based on the detected keywords and emotion recognition results, and a means for storing the detection results and the voice data.

[1387] A way to record audio

[1388] The telephone terminals used by users are equipped with a function to record calls as digital audio data. This function starts recording from the moment the call starts and saves it when the call ends. A specific example is a built-in application on a smartphone that provides a recording function.

[1389] A means of converting recorded audio into text

[1390] When the call ends, the device sends the recorded voice data to the server, which converts the voice data into text using a speech recognition engine (e.g., Google Speech Recognition API). This saves the call contents as text data.

[1391] A means of analyzing the generated text to detect specific keywords

[1392] The server passes the generated text data to a natural language processing (NLP) module, which is responsible for detecting specific keywords and phrases (e.g., "transfer money" or "prepaid card"). If the NLP module detects keywords that are likely to be fraudulent, the process proceeds to the next step.

[1393] A method for recognizing emotions from voice data

[1394] At the same time, the server analyzes the user's emotions using a sentiment analysis engine (e.g., the sentiment analysis model from the transformers library), which identifies emotions (e.g., anxiety, fear) based on the tone, pitch, and intensity of the voice.

[1395] A means of sending notifications based on detected keywords and emotion recognition results

[1396] The server integrates the detected keywords with the results of emotion recognition, and if there is a high possibility of fraud, it sends a notification to the user and their family. The notification includes the specific content of the call and the detected emotion, and contains information urging the user to take prompt action. Notifications can be sent via email or push notification.

[1397] A means of storing detection results and audio data

[1398] The server stores all analysis results and audio data in a database, which can then be reviewed and shared with law enforcement agencies.

[1399] Specific examples

[1400] For example, if an elderly person receives a call from a scammer saying, "I want to buy a prepaid card at a convenience store and please tell me the number," the following process takes place: The call is recorded and converted into text using speech recognition. Next, the NLP module detects the phrase "I want to buy a prepaid card." At the same time, the sentiment analysis engine recognizes the user's anxiety. As a result, the server determines that there is a high possibility of fraud and sends a notification to the user's family saying, "You have received a potentially fraudulent call. The call says, 'I want to buy a prepaid card at a convenience store and please tell me the number.' Please be careful as the user is in an emotionally unstable state."

[1401] Prompt Sentence Examples

[1402] "Perform the following tasks to generate text from the audio data containing keywords that increase suspicion of fraud and detect the user's emotional state (anxiety, fear, etc.). The transcript of the audio data is as follows: "This is my son. I'd like to buy a prepaid card at a convenience store and would like the number." Based on this, assess the likelihood of fraud and the emotional state."

[1403] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1404] Step 1:

[1405] The device detects an incoming call and activates the call recording function. When the call starts, the device automatically starts recording the conversation as digital audio data. The input is the incoming call signal, and the output is the recorded audio data.

[1406] Step 2:

[1407] When the call ends, the device compresses the recorded voice data and sends it to the server. The device transmits the data using a secure protocol over the Internet, maintaining the privacy and security of the data. The input is the recorded voice data, and the output is a data packet containing the compressed voice data.

[1408] Step 3:

[1409] The server passes the received voice data to a speech recognition engine, which converts the voice into text. The speech recognition engine (for example, Google Speech Recognition API) analyzes the voice data and generates corresponding text data. The input is compressed voice data, and the output is text data.

[1410] Step 4:

[1411] The server inputs the generated text data into a natural language processing (NLP) module, which analyzes the text to detect specific keywords and phrases. The NLP module searches for fraud-related keywords (e.g., "transfer" or "prepaid card") within the text and, if detected, proceeds to the next step. The input is the text data, and the output is the detected keyword information.

[1412] Step 5:

[1413] The server inputs the voice data into an emotion recognition engine to analyze the user's emotional state. The emotion recognition engine (e.g., the emotion analysis model in the transformers library) analyzes voice features such as tone, pitch, and intensity to identify the user's emotion (e.g., anxiety, fear). The input is the voice data, and the output is the emotion recognition result.

[1414] Step 6:

[1415] The server integrates the detected keywords and emotion recognition results to comprehensively assess the likelihood of fraud. Based on this result, if there is a high suspicion of fraud, it prepares to send an alert to the user and their family. The input is the keyword detection information and emotion recognition results, and the output is a notification alert message.

[1416] Step 7:

[1417] The server constructs an alert message and sends it to the specified notification recipients (e.g., the user's family). Notification methods include email and push notifications. The input is the alert message for notification, and the output is the sent notification.

[1418] Step 8:

[1419] The server stores all analysis results and audio recordings in a database, allowing them to be reviewed at a later date and shared with law enforcement if necessary. The inputs are analysis results and audio recordings, and the output is the stored data.

[1420] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1421] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1422] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1423] [Fourth embodiment]

[1424] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1425] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1426] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1427] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1428] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1429] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1430] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1431] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1432] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1433] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1434] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1435] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1436] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1437] ---

[1438] The present invention relates to a system for detecting telephone fraud in real time and notifying elderly people and their families. A specific embodiment of the system and program processing will be described below.

[1439] This system detects the contents of phone calls in real time, analyzes specific keywords, and notifies the elderly and their families. The system consists of the following main components:

[1440] 1. Terminal: A telephone terminal used by the elderly. This terminal records calls and generates audio data.

[1441] 2. Server: Receives voice data, performs speech recognition, generates text data, and analyzes and detects keywords and phrases that may be fraudulent.

[1442] 3. Users: Elderly people and their families. They receive notifications from the server and take appropriate action in case of possible fraud.

[1443] Program processing

[1444] 1. Acquiring audio data

[1445] When the device detects an incoming call, it automatically starts recording the call, and when the call ends, it temporarily saves the recorded audio data.

[1446] 2. Sending audio data

[1447] The device compresses the recorded audio data and sends it to the server using a secure protocol.

[1448] 3. Converting voice data to text

[1449] The server passes the received voice data to a voice recognition engine, converts the voice into text, and obtains the text data generated by the voice recognition engine.

[1450] 4. Natural Language Processing for Fraud Detection

[1451] The server analyzes the text data, detecting specific keywords and phrases (e.g., "transfer" or "prepaid card"), and uses natural language processing techniques to evaluate the content for potential fraud.

[1452] 5. Notification Processing

[1453] Based on the analysis results, the server sends a notification to the elderly person and their family, including the specific wording of the suspected fraudulent call and a summary of the call.

[1454] Users can receive notifications, review the details, and take immediate action, for example, allowing family members to contact the elderly and prevent fraud before it occurs.

[1455] 6. Data Storage and Sharing

[1456] The server stores the detection results and audio data in a database, allowing for detailed analysis and review later if necessary.

[1457] Where necessary, we may share collected data with law enforcement agencies to help them investigate fraud trends and patterns.

[1458] Specific examples

[1459] Detection of bank transfer fraud

[1460] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "This is my son. I urgently need money, so please transfer it to the specified account right now."

[1461] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[1462] 3. The server passes the voice data to a voice recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer it to the specified account immediately."

[1463] 4. The server analyzes the text data and detects the phrase "Transfer the money to the specified account." It determines that the message is likely to be fraudulent.

[1464] 5. The server immediately sends a notification to the user (the elderly person's family member) saying, "You have received a potentially fraudulent call. The message is, 'We urgently need money, so please transfer it to the specified account immediately.' Please be careful."

[1465] 6. The user will receive a notification and contact the elderly person to warn them and instruct them not to make the transfer.

[1466] Prepaid card fraud detection

[1467] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "Please buy a prepaid card at a convenience store and give me the number."

[1468] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[1469] 3. The server passes the voice data to a voice recognition engine, which generates text data such as, "Please purchase a prepaid card at a convenience store and tell me the number."

[1470] 4. The server analyzes the text data and detects the phrase "purchase a prepaid card." It determines that the data is likely fraudulent.

[1471] 5. The server immediately sends a notification to the user (the elderly person's family member) saying, "You have received a potentially fraudulent call. The caller asks you to purchase a prepaid card at a convenience store and provide the number. Please be careful."

[1472] 6. The user will receive a notification and contact the elderly person to warn them and provide guidance on how to prevent fraud.

[1473] Through real-time analysis and notification of voice data, this system is an effective means of protecting the safety of the elderly and is useful in combating widespread fraud.

[1474] ---

[1475] That's all.

[1476] The processing flow will be explained below.

[1477] ---

[1478] Step 1:

[1479] When the device detects an incoming call, it automatically starts recording the call, so that the conversation between the elderly person and the other party is recorded as digital audio data from the moment the call begins.

[1480] ---

[1481] Step 2:

[1482] After the call ends, the device compresses the recorded audio data to improve data transfer efficiency and save storage space. The compressed audio data is then temporarily stored on the device.

[1483] ---

[1484] Step 3:

[1485] The device uses a secure protocol to send compressed audio data to the server, using encrypted communications over the internet to ensure data privacy and security.

[1486] ---

[1487] Step 4:

[1488] The server passes the received voice data to a voice recognition engine, which converts the voice into text. The voice recognition engine then analyzes the voice data and outputs it as "character string data."

[1489] ---

[1490] Step 5:

[1491] The server then inputs the generated text data into a natural language processing (NLP) module, which analyzes the text and detects specific keywords and phrases, such as whether it contains terms associated with fraud, such as "transfer" or "prepaid card."

[1492] ---

[1493] Step 6:

[1494] The server evaluates the analysis results from the NLP module and, if it determines that there is a high possibility of fraud, begins preparing a notification. At this stage, the notification content is generated and a message is created to be sent to the elderly person and their family.

[1495] ---

[1496] Step 7:

[1497] The server sends a notification message, which can be sent to the user via SMS, email, or a dedicated app, allowing the user to quickly become aware of potential fraud and take action.

[1498] ---

[1499] Step 8:

[1500] The user receives a notification, which includes the suspected fraudulent text and a summary of the call, and can then contact the senior to provide guidance or advice.

[1501] ---

[1502] Step 9:

[1503] The server stores the analysis results and audio data in a database, which can be used for later review and analysis.

[1504] ---

[1505] Step 10:

[1506] If necessary, the server will initiate the process of sharing collected data with law enforcement agencies to investigate fraud trends and patterns and assist in legal action.

[1507] ---

[1508] That's all.

[1509] Example 1

[1510] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1511] In recent years, telephone fraud has been on the rise, with elderly people being particularly vulnerable. To prevent fraud, it is necessary to detect suspicious calls in real time and promptly notify the elderly person and their family. However, conventional systems are unable to adequately perform this real-time detection and notification, which often results in damage occurring. For this reason, there is a need for a system that can detect and notify fraud more effectively and quickly.

[1512] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1513] In this invention, the server includes a means for compressing voice data and transmitting it using a secure protocol, a means for converting recorded voice into text, a natural language processing means for analyzing the generated text and detecting specific keywords, a means for sending a notification based on the detected keywords, a means for sending the notification to the user, a means for saving the analysis results and the voice data, and a means for sharing the analysis results and the voice data with law enforcement agencies as needed. This makes it possible to detect suspected fraudulent calls in real time and quickly notify the elderly person and their family.

[1514] "Audio recording means" refers to a device or program that automatically records telephone conversations and saves the audio data.

[1515] "Means for compressing recorded audio data and transmitting it using a secure protocol" refers to a device or program that compresses recorded audio files using a compression algorithm and transmits them to a server using a secure communication protocol such as HTTPS or SSL.

[1516] A "means for converting recorded speech to text" is a system or program that uses a speech recognition engine to analyze speech data and convert it into a human-understandable text format.

[1517] "Natural language processing means for analyzing generated text and detecting specific keywords" means a system or program that uses natural language processing techniques to analyze text data and detect specific keywords or phrases that may be suspected of being fraudulent.

[1518] "Means for sending notifications based on detected keywords" means a system or program for sending warnings or notifications to users when detected keywords or phrases indicate suspected fraud.

[1519] "Means for sending notifications to users" means a system or program for sending notifications to the elderly person and their family members via SMS, email, or other communication means.

[1520] "Means for storing analysis results and audio data" refers to a system or program for storing analysis results and audio data in a database or file system, and keeping them in a state where they can be accessed and analyzed later.

[1521] "Means for sharing analytics results and audio data with law enforcement as needed" means a system or program that shares stored analytics results and audio data with law enforcement to assist in fraud investigation and prevention.

[1522] "Communication terminals used by the elderly" refer to telephones, smartphones, or other communication devices capable of making calls that are primarily used by the elderly in their daily lives.

[1523] A "server with a speech recognition engine" is a server that has speech recognition capabilities to convert received speech data into text, and is a device that has the computing resources to execute programs and algorithms to analyze speech.

[1524] The present invention relates to a system for detecting telephone fraud in real time and notifying elderly people and their families of the fraud. Specific embodiments of the system will be described below.

[1525] This system consists of the following main hardware and software:

[1526] Hardware and Software Configuration

[1527] 1. Device:

[1528] Telephones and smartphones are used as communication devices by the elderly. These devices record calls and generate audio data.

[1529] The software used is a module for recording (for example, a recording class developed in Java or Swift).

[1530] 2. Server:

[1531] A server that receives voice data, performs speech recognition, and generates text data. The server is equipped with a speech recognition engine (e.g., Google Speech-to-Text API) and a natural language processing engine (e.g., SpaCy or NLTK).

[1532] To run the program, web frameworks such as Python, Flask, and Django are used.

[1533] 3. User:

[1534] The elderly person and their family members receive notifications from the server and take appropriate action if there is a possibility of fraud.

[1535] Data processing and calculation

[1536] Acquiring audio data

[1537] When the device detects an incoming call, it automatically starts recording the call. For example, you can use the smartphone's built-in app or a dedicated recording app. When the call ends, the recorded audio data is temporarily stored on the device.

[1538] Sending audio data

[1539] The device compresses the recorded audio data and sends it to the server using a secure protocol (e.g., HTTPS or SSL) using an algorithm such as GZIP.

[1540] Converting audio data to text

[1541] The server passes the received voice data to a speech recognition engine (e.g., Google Speech-to-Text API) and converts the voice into text, which is then stored in a database.

[1542] Natural Language Processing for Fraud Detection

[1543] The server analyzes the text data using a natural language processing engine (e.g., SpaCy or NLTK) to detect specific keywords and phrases (e.g., "transfer," "prepaid card," etc.).

[1544] Notification Processing

[1545] Based on the analysis results, the server sends a notification to the elderly person and their family, including specific wording of the suspected fraud and a summary of the call. For example, notifications are sent via SMS or email using Twilio or SendGrid.

[1546] Data storage and sharing

[1547] The server stores the detection results and audio data in a database using MySQL or PostgreSQL. The stored data can be analyzed and reviewed in detail at a later date, and can be shared with law enforcement agencies if necessary.

[1548] Specific examples

[1549] Detection of bank transfer fraud

[1550] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "This is my son. I urgently need money, so please transfer it to the specified account right now."

[1551] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[1552] 3. The server passes the voice data to a voice recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer it to the specified account immediately."

[1553] 4. The server analyzes the text data and detects the phrase "Transfer the money to the specified account." It determines that the message is likely to be fraudulent.

[1554] 5. The server immediately sends a notification to the user (the elderly person's family member) saying, "You have received a potentially fraudulent call. The message is, 'We urgently need money, so please transfer it to the specified account immediately.' Please be careful."

[1555] 6. The user receives a notification and contacts the elderly person to warn them and prevent any damage from occurring.

[1556] Prepaid card fraud detection

[1557] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "Please buy a prepaid card at a convenience store and give me the number."

[1558] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[1559] 3. The server passes the voice data to a voice recognition engine, which generates text data such as, "Please purchase a prepaid card at a convenience store and tell me the number."

[1560] 4. The server analyzes the text data and detects the phrase "purchase a prepaid card." It determines that the data is likely fraudulent.

[1561] 5. The server immediately sends a notification to the user (the elderly person's family member) saying, "You have received a potentially fraudulent call. The caller asks you to purchase a prepaid card at a convenience store and provide the number. Please be careful."

[1562] 6. The user receives a notification and contacts the elderly person to warn them and prevent any damage from occurring.

[1563] Prompt Sentence Examples

[1564] In the case of detecting bank transfer fraud

[1565] "Please tell me how to build a system that notifies the elderly person and their family when they receive a call from someone pretending to be their son requesting that they transfer money to a specified account."

[1566] Prepaid card fraud detection

[1567] "Please tell me how to build a system that notifies elderly people and their families when they purchase a prepaid card at a convenience store and are asked to provide the card number."

[1568] The system provides real-time voice analysis and rapid notification to protect against fraud targeting seniors, making it effective in combating a wide range of frauds.

[1569] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1570] Step 1: Acquire audio data

[1571] When the device detects an incoming call, it automatically starts recording the call, recording the contents of the call in real time.

[1572] Input: Call audio

[1573] Data processing: Save call audio as a recording file

[1574] Output: Recording file

[1575] How it works: The device's calling app triggers an incoming call event, calls a recording module (for example, a recording class written in Java or Swift) to start recording, and then saves the recording file in the device's experience data storage after the call ends.

[1576] Step 2: Sending audio data

[1577] The device compresses the recorded audio data and sends it to the server using a secure protocol.

[1578] Input: Recording file

[1579] Data processing: Compress the audio data using GZIP or other methods and package it for HTTPS requests

[1580] Output: Compressed audio data is sent to the server

[1581] What it does: It compresses the recorded audio data using the GZIP compression algorithm and sends it to the server's API endpoint using the HTTPS protocol, for example, using Python's requests library or Java's HttpURLConnection class.

[1582] Step 3: Convert audio data to text

[1583] The server passes the received voice data to a voice recognition engine, which converts the voice into text.

[1584] Input: Compressed audio data

[1585] Data processing: Converting speech to text using a speech recognition engine (such as Google Speech-to-Text API)

[1586] Output: Text data

[1587] What happens: The server receives the audio data, decompresses it, and passes it to the speech recognition engine. The speech recognition engine analyzes the audio file and generates text data, which is then stored in a database.

[1588] Step 4: Natural Language Processing for Fraud Detection

[1589] The server analyzes the text data using a natural language processing engine to detect specific keywords and phrases.

[1590] Input: Text data

[1591] Data processing: Use natural language processing engines (such as SpaCy or NLTK) to extract and analyze specific keywords and phrases within text data.

[1592] Output: Analysis results showing suspected fraudulent phrases and keywords

[1593] How it works: The server passes the text data to a natural language processing engine, which analyzes it for potentially fraudulent keywords and phrases. Based on the analysis, any data that is suspected to be fraudulent is flagged and stored in a database.

[1594] Step 5: Notification Processing

[1595] Based on the analysis results, the server sends a notification to the elderly person and their family.

[1596] Input: Analysis results

[1597] Data processing: Generate notification content and send it using a notification service such as Twilio or SendGrid

[1598] Output: Email and SMS notifications sent to the elderly person and their family

[1599] Specific operation: The server composes the analysis results in JSON format and sends emails or SMS to the elderly person and their family members through existing notification services such as Twilio and SendGrid.

[1600] Step 6: Data storage and sharing

[1601] The server stores the detection results and audio data in a database, which may be shared with law enforcement agencies if necessary.

[1602] Input: Analysis results and audio data

[1603] Data processing: Analysis results and audio data are written to a database and shared with law enforcement agencies as needed

[1604] Output: Analysis results and audio data stored in a database

[1605] What it does: The server uses an ORM (Object-Relational Mapping) tool such as SQLAlchemy to write the analysis results and audio data to a database (such as MySQL or PostgreSQL), and provides access and data sharing to law enforcement agencies as needed.

[1606] These steps will allow the entire system to work together to protect seniors from phone scams.

[1607] (Application example 1)

[1608] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1609] In recent years, frauds targeting the elderly have been increasing. However, many elderly people do not have sufficient means to avoid fraud, and the damage caused by these frauds is becoming more serious. To solve this problem, a system is needed that can detect telephone frauds in real time and quickly notify the elderly and their families.

[1610] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1611] In this invention, the server includes a means for transmitting recorded voice data to a remote server using a security protocol, a means for converting the voice data into text using a voice recognition engine, a means for analyzing the text data using natural language processing technology, and a means for sending a push notification to a smartphone, thereby enabling real-time detection of suspected fraud and prompt notification to the elderly person and their family.

[1612] "Audio recording means" means a device or function that records the contents of a telephone conversation as digital audio data.

[1613] "Means for converting recorded voice into text" refers to a technology for converting voice data into text in a natural language, i.e., a device or function that generates text data from voice using a voice recognition engine.

[1614] "Means for analyzing the generated text and detecting specific keywords" refers to a device or function that analyzes the generated text data using algorithms or natural language processing techniques to identify keywords or phrases that may be fraudulent.

[1615] A "means for sending notifications based on detected keywords" is a device or function that, when a keyword that indicates suspected fraud is detected, notifies the user or their family of the potential fraud via a notification mechanism.

[1616] "Means for storing detection results and audio data" refers to a device or function that stores the analyzed text data and original audio data in a database or storage system so that they can be accessed later.

[1617] "Means for transmitting recorded audio data to a remote server using a secure protocol" refers to a device or function that compresses the recorded audio data and transfers it to a remote server using a secure communication protocol (e.g., HTTPS).

[1618] "Means for converting voice data into text using a voice recognition engine" refers to a device or function that mechanically converts voice data into text using voice recognition software or a cloud service.

[1619] A "means for analyzing text data using natural language processing techniques" is a device or function that uses natural language processing algorithms or models to evaluate text data and identify keywords or phrases that may contain indicators of fraud.

[1620] A "means for sending push notifications to a smartphone" is a device or system with push notification functionality for quickly notifying users and their families when potential fraud is detected.

[1621] The present invention relates to a system for detecting telephone fraud in real time and notifying elderly people and their families promptly. The following describes a specific embodiment and the hardware and software used.

[1622] System configuration

[1623] The system consists of the following main components:

[1624] 1. Device:

[1625] This is a telephone terminal used by elderly people. It records calls and generates audio data.

[1626] This includes communication devices such as smartphones with call recording functionality installed.

[1627] 2. Server:

[1628] Voice data is received, voice recognition is performed, and text data is generated.

[1629] Specific keywords are analyzed using natural language processing technology.

[1630] Google Cloud Speech-to-Text and Amazon Transcribe are used as speech recognition engines.

[1631] Natural language processing technologies used include Google Cloud Natural Language API and Amazon Comprehend.

[1632] 3. User:

[1633] The elderly themselves and their families.

[1634] Receive notifications from the server and take appropriate action in case of potential fraud.

[1635] About program processing

[1636] The processing flow by the system is shown below.

[1637] 1. Acquire audio data:

[1638] When the device detects an incoming call, it automatically records the call. When the call ends, it stops recording and temporarily saves the audio data.

[1639] 2. Sending audio data:

[1640] The device compresses the recorded audio data and sends it to the server using a secure communication protocol (e.g., HTTPS).

[1641] 3. Audio to text conversion:

[1642] The server passes the received voice data to a speech recognition engine, converts the voice into text data, and generates highly accurate text data from the voice data using Google Cloud Speech-to-Text or Amazon Transcribe.

[1643] 4. Natural Language Processing for Fraud Detection:

[1644] The server analyzes the text data using natural language processing technology (Google Cloud Natural Language API and Amazon Comprehend) to detect keywords and phrases that may be fraudulent.

[1645] 5. Notification Processing:

[1646] Based on the analysis results, the server sends a push notification to the smartphone, and also notifies the user and their family members via email and SMS.

[1647] Use Firebase Cloud Messaging or Amazon SNS for push notification.

[1648] 6. Data Storage and Sharing:

[1649] The server stores the analysis results and audio data in a cloud database (Firebase Firestore or Amazon DynamoDB), which allows the information to be shared with law enforcement agencies if necessary.

[1650] Specific examples

[1651] Example of detecting bank transfer fraud

[1652] The device detects an incoming call and records the conversation. The scammer says, "This is my son. I urgently need money, so please transfer the money to my specified account now." When the call ends, the recorded audio data is compressed and sent to the server. The server passes the audio data to a speech recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer the money to my specified account now." The server analyzes the text data and detects the phrase "transfer the money to my specified account." It immediately sends a notification to the user saying, "You have received a potentially fraudulent call. The message is, 'I urgently need money, so please transfer the money to my specified account now.' Please be careful."

[1653] Prepaid card fraud detection example

[1654] The device detects an incoming call and records the conversation. The scammer says, "Please buy a prepaid card at a convenience store and give me the number." When the call ends, the recorded audio data is compressed and sent to the server. The server passes the audio data to a speech recognition engine, which generates text data saying, "Please buy a prepaid card at a convenience store and give me the number." The server analyzes the text data and detects the phrase "please buy a prepaid card." It immediately sends a notification to the user saying, "You have received a potentially fraudulent call. The message is, 'Please buy a prepaid card at a convenience store and give me the number.' Please be careful."

[1655] In this way, this system can protect the elderly from fraud through real-time analysis and notification of voice data.

[1656] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1657] Step 1:

[1658] The device detects an incoming call and automatically starts recording the call. When the call ends, the recording stops and the audio data is temporarily saved.

[1659] Specifically, the call recording function installed on the smartphone is used to save the audio data during the call in a digital format, such as a .wav or .3gp file.

[1660] Step 2:

[1661] The device compresses the recorded audio data and sends it to the server using a secure communication protocol (e.g., HTTPS). The audio data is compressed to transmit the audio file efficiently using less bandwidth.

[1662] Input: Recorded audio data

[1663] Output: Compressed audio data file

[1664] Specifically, the recorded audio data is converted into a compressed format such as .mp3 or .opus and uploaded to a server using the HTTPS protocol.

[1665] Step 3:

[1666] The server passes the received voice data to a voice recognition engine, which converts the voice into text.

[1667] Input: Compressed audio data file

[1668] Output: Text data

[1669] Specifically, the voice data is converted into text data using the speech recognition engines of Google Cloud Speech-to-Text and Amazon Transcribe. The server calls the API to send the audio file and obtains the returned text data.

[1670] Step 4:

[1671] The server analyzes the text data using natural language processing technology to detect specific keywords and phrases.

[1672] Input: Text data

[1673] Output: Analysis results containing keywords and phrases that may be fraudulent

[1674] Specifically, the system analyzes text data using Google Cloud Natural Language API and Amazon Comprehend to detect fraudulent keywords such as "transfer" and "prepaid card."

[1675] Step 5:

[1676] Based on the analysis results, the server sends a push notification to the smartphone, and also notifies the user and their family via email or SMS.

[1677] Input: Suspected fraud analysis results

[1678] Output: Push notification and email / SMS notification

[1679] Specifically, it uses Firebase Cloud Messaging and Amazon SNS to send push notifications to smartphones, as well as email and SMS notifications to users and their families, containing specific details about the suspected scam and a warning message.

[1680] Step 6:

[1681] The server stores the detection results and audio data in a cloud database (such as Firebase Firestore or Amazon DynamoDB) and shares the data with law enforcement agencies if necessary.

[1682] Input: Analysis results and audio data

[1683] Output: Data stored in a cloud database

[1684] Specifically, the system securely transmits and stores voice data and analysis results in a cloud database, and provides authenticated access for data sharing with law enforcement agencies.

[1685] In this way, the individual processing steps work together to effectively operate a system that protects the elderly from fraud.

[1686] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1687] ---

[1688] This invention relates to a system that detects telephone fraud in real time and notifies the elderly person and their family. In particular, by combining it with an emotion engine that recognizes the user's emotions from their voice, the possibility of fraud can be further increased. This system is implemented with the following components.

[1689] This system detects the content of phone conversations in real time, analyzes specific keywords, recognizes the user's emotions, and notifies the elderly and their families.The system consists of the following main components.

[1690] 1. Terminal: A telephone terminal used by the elderly. This terminal records calls and generates audio data.

[1691] 2. Server: Receives voice data, performs speech recognition, generates text data, analyzes keywords and phrases that may be fraudulent, and recognizes user emotions.

[1692] 3. Emotion Engine: Recognizes user emotions from recorded voice data. Emotion recognition analyzes the emotional tone of a call to assess the likelihood of fraud.

[1693] 4. Users: Elderly people and their families. They receive notifications from the server and take appropriate action in case of possible fraud.

[1694] Program processing

[1695] 1. Acquiring audio data

[1696] When the device detects an incoming call, it automatically starts recording the call, so that the conversation between the elderly person and the other party is recorded as digital audio data from the moment the call begins.

[1697] 2. Sending audio data

[1698] After the call ends, the device compresses the recorded audio data to improve data transfer efficiency and save storage space. The compressed audio data is then temporarily stored on the device.

[1699] The device uses a secure protocol to send compressed audio data to the server, using encrypted communications over the internet to ensure data privacy and security.

[1700] 3. Converting voice data to text

[1701] The server passes the received voice data to a voice recognition engine, which converts the voice into text. The voice recognition engine then analyzes the voice data and outputs it as "character string data."

[1702] 4. Natural Language Processing for Fraud Detection

[1703] The server then inputs the generated text data into a natural language processing (NLP) module, which analyzes the text and detects specific keywords and phrases, such as whether it contains terms associated with fraud, such as "transfer" or "prepaid card."

[1704] 5. Emotion recognition

[1705] The server inputs the voice data into the emotion engine and analyzes the user's emotions. The emotion engine recognizes the user's emotions (e.g., anxiety, tension, fear) from the tone, pitch, and intensity of the voice.

[1706] The server integrates the emotion recognition results with the NLP results to provide a comprehensive assessment of the likelihood of fraud.

[1707] 6. Notification Processing

[1708] Based on the analysis results, the server sends a notification to the elderly person and their family, including the specific wording of the suspected fraudulent call and a summary of the call.

[1709] The server then adjusts the notification content based on the results of emotion recognition. For example, if the user's emotions are very unstable, the notification will include an alert urging immediate action.

[1710] 7. User Response

[1711] The user receives a notification and checks its contents, which include the suspected fraudulent text, a summary of the call, and the results of emotion recognition. The user can then contact the elderly person and take action if necessary.

[1712] 8. Data Storage and Sharing

[1713] The server stores the analysis results and audio data in a database, which can be used for later review and analysis.

[1714] Where necessary, we may share collected data with law enforcement agencies to help them investigate fraudulent trends and patterns.

[1715] Specific examples

[1716] Detection of bank transfer fraud and emotion recognition

[1717] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "This is my son. I urgently need money, so please transfer it to the specified account right now."

[1718] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[1719] 3. The server passes the voice data to a voice recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer it to the specified account immediately."

[1720] 4. The server analyzes the text data and detects the phrase "Transfer the money to the specified account." It determines that the message is likely to be fraudulent.

[1721] 5. The server passes the voice data to an emotion engine to recognize the user's emotions (e.g., nervousness, anxiety). The recognition results are evaluated as factors that further increase the likelihood of fraud.

[1722] 6. The server immediately sends a notification to the user (family member of the elderly person) saying, "You have received a potentially fraudulent call. The message is, 'We urgently need money, so please transfer it to the specified account immediately.' Please be careful as the user's emotions are unstable."

[1723] 7. The user will receive a notification and contact the elderly person to warn them and instruct them not to make the transfer.

[1724] Prepaid Card Fraud Detection and Emotion Recognition

[1725] 1. The device detects an incoming call and records the conversation. For example, a scammer might say, "Please buy a prepaid card at a convenience store and give me the number."

[1726] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[1727] 3. The server passes the voice data to a voice recognition engine, which generates text data such as, "Please purchase a prepaid card at a convenience store and tell me the number."

[1728] 4. The server analyzes the text data and detects the phrase "purchase a prepaid card." It determines that the data is likely fraudulent.

[1729] 5. The server passes the voice data to an emotion engine to recognize the user's emotion (e.g., anxiety, alarm). The recognition result is evaluated as a factor that further increases the likelihood of fraud.

[1730] 6. The server immediately sends a notification to the user (family member of the elderly person) saying, "You have received a potentially fraudulent call. The caller asks you to purchase a prepaid card at a convenience store and provide the number. Please be careful as the user's emotions are unstable."

[1731] 7. The user will receive a notification and contact the elderly person to warn them and provide guidance on how to prevent fraud.

[1732] Through real-time analysis of voice data and emotion recognition, this system is an effective means of protecting the safety of the elderly and can be useful in combating widespread fraud.

[1733] ---

[1734] That's all.

[1735] The processing flow will be explained below.

[1736] ---

[1737] Step 1:

[1738] When the device detects an incoming call, it automatically starts recording the call and continues recording until the call ends, saving the conversation as digital audio data.

[1739] ---

[1740] Step 2:

[1741] After the call ends, the device compresses the recorded audio data, which saves storage space and improves data transmission efficiency.

[1742] ---

[1743] Step 3:

[1744] The device uses a secure protocol to send compressed audio data to the server, where it is encrypted and securely transmitted over the internet.

[1745] ---

[1746] Step 4:

[1747] The server passes the received voice data to a speech recognition engine, which converts the voice into text. The speech recognition engine analyzes the voice data and generates corresponding text data.

[1748] ---

[1749] Step 5:

[1750] The server inputs the generated text data into a natural language processing (NLP) module, which analyzes the text data and detects specific keywords and phrases related to fraud (e.g., "transfer money" or "prepaid card").

[1751] ---

[1752] Step 6:

[1753] The server passes the voice data to the emotion engine to recognize the user's emotion. The emotion engine analyzes the voice tone, pitch, intensity, etc. to identify the user's emotion (e.g., anxiety, tension, fear, etc.).

[1754] ---

[1755] Step 7:

[1756] The server combines the analysis results of the NLP module and the recognition results of the emotion engine to comprehensively evaluate the likelihood of fraud. If the emotion engine's results increase suspicion of fraud, the evaluation is revised upward.

[1757] ---

[1758] Step 8:

[1759] If the server determines that fraud is likely, it prepares a notification message containing the suspected fraudulent text detected, a summary of the call, and the results of emotion recognition.

[1760] ---

[1761] Step 9:

[1762] The server sends a notification message to the user (elderly person and their family) via SMS, email, or a dedicated app.

[1763] ---

[1764] Step 10:

[1765] The user receives and reviews the notification, which includes the specific suspected fraudulent text, a summary of the call, and the results of emotion recognition. The user then contacts the senior and takes appropriate action to address the potential fraud.

[1766] ---

[1767] Step 11:

[1768] The server stores the analysis results and audio data in a database, making it easy to check and analyze later.

[1769] ---

[1770] Step 12:

[1771] If necessary, the server will initiate the process of sharing collected data with law enforcement agencies to investigate fraudulent trends and patterns and assist in legal action.

[1772] ---

[1773] That's all.

[1774] Example 2

[1775] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1776] Conventional telephone fraud prevention systems are limited in their ability to record and analyze call content, making it difficult to detect fraud in real time. Furthermore, they are unable to take into account the emotions of the elderly, such as anxiety and tension, which can lead to reduced accuracy in detecting fraudulent activity. Furthermore, notification functions are inadequate, leaving a lack of means to quickly alert the elderly and their families. There is a need for a new system that can solve these problems.

[1777] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes a means for recording voice, a means for compressing and transmitting the recorded voice, a means for converting the transmitted voice into text using a voice recognition engine, a means for analyzing the generated text using a natural language processing module and detecting specific keywords, a means for recognizing the user's emotions from the voice data, a means for comprehensively evaluating the possibility of fraud, a means for sending a notification based on the evaluation result, and a means for saving the detection result and the voice data. This makes it possible to analyze the content of the call and the user's emotions in real time, identify the possibility of fraud with high accuracy, and promptly notify the elderly person and their family.

[1778] A "voice recording means" is a device capable of recording voice data in digital format at the initiation of a telephone call.

[1779] The "means for compressing and transmitting recorded audio" is a device that has the function of compressing recorded audio data in a manner that reduces the data volume and transmitting it to a server using a secure protocol.

[1780] The "means for converting transmitted voice into text using a voice recognition engine" refers to software or hardware for analyzing received voice data on a server and converting it into corresponding text data.

[1781] "Means for analyzing the generated text using a natural language processing module and detecting specific keywords" refers to a software module that has the function of analyzing text data using natural language processing technology and detecting specific keywords and phrases related to fraud.

[1782] "Means for recognizing user emotions from voice data" refers to a system or software that analyzes the tone, pitch, intensity, etc. of recorded voice data to recognize the user's emotional state (e.g., anxiety, tension, fear).

[1783] "Means for comprehensively assessing the likelihood of fraud" refers to algorithms or systems that integrate the results of keyword detection using natural language processing and emotion recognition to determine the likelihood of fraud.

[1784] The "means for sending notifications based on the evaluation results" refers to a system that has the function of sending notifications containing warnings and detailed information to the elderly person and their family if it is determined that there is a high possibility of fraud.

[1785] The "means for storing the detection results and audio data" refers to a device or system that has the function of storing the analysis results and recorded audio data in storage such as a database for later confirmation or analysis.

[1786] This invention relates to a system that detects telephone fraud in real time and notifies the elderly person and their family. The main purpose of this system is to record voice, analyze the recorded voice, and evaluate the possibility of fraud with high accuracy. In particular, by incorporating a mechanism to recognize the user's emotions from voice data, the accuracy of fraud detection can be further improved. This system is realized using the following specific hardware and software.

[1787] Hardware and Software Configuration

[1788] Terminal

[1789] The telephone terminals used by the elderly have the ability to record phone conversations in real time. The terminals record conversations as digital audio data from the moment the call begins until it ends. The recorded audio data is compressed for efficient storage and sent to a server using a secure protocol (e.g., HTTPS).

[1790] server

[1791] The server is the central location for analyzing the received audio data. It contains the following main software modules:

[1792] Speech recognition engine: Converts received voice data into text data. Google Cloud Speech-to-Text and IBM Watson Speech to Text are available as this engine.

[1793] Natural Language Processing (NLP) module: Analyzes the generated text data and detects specific keywords and phrases related to fraud. This module is powered by IBM Watson Natural Language Understanding.

[1794] Emotion engine: Used to recognize user emotions from recorded voice data. Tools such as Affectiva and Microsoft Azure Emotion Recognition are available.

[1795] Database: Analysis results and audio data are stored. The stored data is used for later review and analysis.

[1796] User

[1797] The users are the elderly themselves and their families. If the server determines that there is a possibility of fraud, the system receives a notification and takes appropriate action.

[1798] Specific examples

[1799] Detection of bank transfer fraud and emotion recognition

[1800] 1. The device detects incoming calls and records the conversation. For example, a scammer might say, "This is my son. I urgently need money, so please transfer it to a specified account right now."

[1801] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[1802] 3. The server passes the voice data to a voice recognition engine, which generates text data saying, "This is my son. I urgently need money, so please transfer it to the specified account immediately."

[1803] 4. The server analyzes the text data and detects the phrase "Please transfer the money to the specified account." It determines that the data is likely to be fraudulent.

[1804] 5. The server passes the voice data to an emotion engine to recognize the user's emotions (e.g., tension, anxiety). The recognition results are used as factors to further increase the likelihood of fraud.

[1805] 6. The server immediately notifies the user (family member of the elderly person) that "You have received a potentially fraudulent call. The caller says, 'We urgently need money, so please transfer the money to the specified account immediately.' Please be careful as the user's emotions are unstable."

[1806] 7. The user will receive a notification and contact the elderly person to warn them and instruct them not to make the transfer.

[1807] Prepaid Card Fraud Detection and Emotion Recognition

[1808] 1. The device detects an incoming call and records the call. For example, a scammer might ask you to buy a prepaid card at a convenience store and provide the number.

[1809] 2. When the call ends, the device compresses the recorded audio data and sends it to the server.

[1810] 3. The server passes the voice data to a voice recognition engine, which generates text data such as, "Please purchase a prepaid card at a convenience store and tell me the number."

[1811] 4. The server analyzes the text data and detects the phrase "purchase a prepaid card." It determines that the data is likely fraudulent.

[1812] 5. The server passes the voice data to an emotion engine to recognize the user's emotion (e.g., anxiety, vigilance). The recognition result is used as a factor to further increase the likelihood of fraud.

[1813] 6. The server immediately notifies the user (family member of the elderly person) that "You have received a potentially fraudulent call. The caller asks you to purchase a prepaid card at a convenience store and provide the number. Please be careful as the user's emotions may be unstable."

[1814] 7. The user will receive a notification and contact the elderly person to warn them and provide guidance on how to prevent fraud.

[1815] Through real-time analysis of voice data and emotion recognition, the system is an effective means of protecting the safety of the elderly and can help combat widespread fraud.

[1816] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1817] Step 1:

[1818] When the device detects an incoming call, it automatically starts recording the call. It receives a call start signal as input. Based on this, it converts the voice data during the call into a digital format in real time and temporarily stores the data. Specifically, the recording process starts the moment the phone receiver is lifted, and the display shows "Recording." The output is the recorded digital voice data.

[1819] Step 2:

[1820] When a call ends, the device compresses the recorded voice data. It receives the digital voice data of the ended call as input. Based on this data, it reduces the data size using voice compression techniques such as High Efficiency Audio Codec (HE-AAC). Specifically, the moment the call ends, the message "Compressing voice data..." is displayed. The compressed voice data is temporarily stored on the device. The output is compressed voice data.

[1821] Step 3:

[1822] The device uses a secure protocol (e.g. HTTPS) to send compressed audio data to the server. As input, it receives compressed audio data. Based on this data, it uses encrypted communications to send it to the server while maintaining the privacy and security of the data. Specifically, it displays "Sending data..." and then displays "Transmission complete" once the transmission is complete. The output is the audio data sent to the server.

[1823] Step 4:

[1824] The server passes the received voice data to a voice recognition engine, which converts the voice into text. Compressed voice data is received as input. Based on this, the voice recognition engine (e.g., Google Cloud Speech-to-Text) analyzes the voice data and generates corresponding text data. Specifically, a progress bar appears saying "Analyzing voice data..." as the analysis progresses. The output is the generated text data.

[1825] Step 5:

[1826] The server inputs the generated text data into a natural language processing (NLP) module to detect specific keywords and phrases. As input, it receives text data from a speech recognition engine. This data is analyzed using a natural language processing module (e.g., IBM Watson Natural Language Understanding) to detect keywords related to fraud (e.g., "transfer" and "prepaid card"). Specifically, it displays "Text analysis in progress..." and, if possible fraud is detected, displays "Suspected fraud." The output is a flag or list of the detection results.

[1827] Step 6:

[1828] The server inputs the voice data into an emotion engine to analyze the user's emotions. Recorded voice data is received as input. This data is analyzed by an emotion engine (e.g., Affectiva or Microsoft Azure Emotion Recognition) to recognize the user's emotions (e.g., anxiety, tension, fear) from the tone, pitch, and intensity of the voice. Specifically, the system displays "Analyzing emotions..." and outputs information such as "User emotion: tension." The output is the recognized emotion data.

[1829] Step 7:

[1830] The server integrates the emotion recognition results with the NLP results to comprehensively assess the likelihood of fraud. As input, it receives the detection results of the natural language processing module and the recognition results of the emotion engine. Based on this data, the algorithm evaluates the likelihood of fraud. Specifically, it displays "Evaluating..." and, if the result indicates a high likelihood of fraud, it displays "High likelihood of fraud." The output is a fraud likelihood flag as the evaluation result.

[1831] Step 8:

[1832] The server sends a notification to the elderly person and their family based on the evaluation results. The server receives the fraud possibility evaluation results as input. Based on this data, it generates the notification content and sends it to the elderly person and their family. The specific operation is to display "Sending notification..." and include specific wording of the suspected fraud and a summary of the call. The output is a message indicating that the notification has been sent.

[1833] Step 9:

[1834] The server saves the analysis results and audio data in a database. It receives the evaluation results and audio data as input. Based on this data, it processes the data for storage in the database. Specifically, it displays "Saving data..." and then displays "Saving completed" once the data has been saved. The output is a confirmation message that the data has been saved.

[1835] Step 10:

[1836] The user receives the notification, checks its contents, and contacts the elderly person to take appropriate action. The input is a notification from the server. This notification includes the specific wording of the suspected fraud, a summary of the call, and the results of emotion recognition. The specific action is to check the notification and contact the elderly person to warn them. The output is a message from the user indicating that the action has been taken.

[1837] (Application example 2)

[1838] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1839] Telephone fraud targeting the elderly is increasing year by year, and the methods are becoming more sophisticated. The elderly are particularly vulnerable to methods that create anxiety and fear, making it difficult to prevent fraud damage before it occurs. The purpose of this invention is to provide a system that detects telephone fraud in real time and promptly notifies the elderly and their families, thereby preventing fraud damage.

[1840] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for recording voice, means for converting the recorded voice into text, means for analyzing the generated text and detecting specific keywords, means for recognizing emotions from voice data, means for sending notifications based on the detected keywords and emotion recognition results, and means for saving the detection results and voice data. This makes it possible to detect fraud-related keywords and changes in emotions contained in the content of calls in real time and to notify the elderly person and their family members promptly and appropriately.

[1841] A "voice recording device" is a device or component that has the ability to record telephone conversations as digital audio data.

[1842] The "means for converting recorded voice to text" is a voice recognition engine or software that converts recorded voice data into character string data.

[1843] The "means for analyzing the generated text and detecting specific keywords" refers to a natural language processing (NLP) module and analysis algorithms that identify specific terms and phrases within the text data.

[1844] The "means for recognizing emotions from voice data" is an emotion analysis engine that analyzes voice characteristics such as tone, pitch, and intensity to recognize the emotional state of the caller.

[1845] The "means for sending notifications based on detected keywords and emotion recognition results" refers to a communication module or notification system that sends alerts to users and their families in the event of suspected fraud.

[1846] "Means for storing detection results and audio data" refers to a database or storage system that stores the analyzed results and recorded audio data in digital format.

[1847] The present invention relates to a system for monitoring telephone calls in real time and notifying seniors and their families of possible fraud. The system includes a means for recording voice, a means for converting the recorded voice into text, a means for analyzing the generated text and detecting specific keywords, a means for recognizing emotions from the voice data, a means for sending notifications based on the detected keywords and emotion recognition results, and a means for storing the detection results and the voice data.

[1848] A way to record audio

[1849] The telephone terminals used by users are equipped with a function to record calls as digital audio data. This function starts recording from the moment the call starts and saves it when the call ends. A specific example is a built-in application on a smartphone that provides a recording function.

[1850] A means of converting recorded audio into text

[1851] When the call ends, the device sends the recorded voice data to the server, which converts the voice data into text using a speech recognition engine (e.g., Google Speech Recognition API). This saves the call contents as text data.

[1852] A means of analyzing the generated text to detect specific keywords

[1853] The server passes the generated text data to a natural language processing (NLP) module, which is responsible for detecting specific keywords and phrases (e.g., "transfer money" or "prepaid card"). If the NLP module detects keywords that are likely to be fraudulent, the process proceeds to the next step.

[1854] A method for recognizing emotions from voice data

[1855] At the same time, the server analyzes the user's emotions using a sentiment analysis engine (e.g., the sentiment analysis model from the transformers library), which identifies emotions (e.g., anxiety, fear) based on the tone, pitch, and intensity of the voice.

[1856] A means of sending notifications based on detected keywords and emotion recognition results

[1857] The server integrates the detected keywords with the results of emotion recognition, and if there is a high possibility of fraud, it sends a notification to the user and their family. The notification includes the specific content of the call and the detected emotion, and contains information urging the user to take prompt action. Notifications can be sent via email or push notification.

[1858] A means of storing detection results and audio data

[1859] The server stores all analysis results and audio data in a database, which can then be reviewed and shared with law enforcement agencies.

[1860] Specific examples

[1861] For example, if an elderly person receives a call from a scammer saying, "I want to buy a prepaid card at a convenience store and please tell me the number," the following process takes place: The call is recorded and converted into text using speech recognition. Next, the NLP module detects the phrase "I want to buy a prepaid card." At the same time, the sentiment analysis engine recognizes the user's anxiety. As a result, the server determines that there is a high possibility of fraud and sends a notification to the user's family saying, "You have received a potentially fraudulent call. The call says, 'I want to buy a prepaid card at a convenience store and please tell me the number.' Please be careful as the user is in an emotionally unstable state."

[1862] Prompt Sentence Examples

[1863] "Perform the following tasks to generate text from the audio data containing keywords that increase suspicion of fraud and detect the user's emotional state (anxiety, fear, etc.). The transcript of the audio data is as follows: "This is my son. I'd like to buy a prepaid card at a convenience store and would like the number." Based on this, assess the likelihood of fraud and the emotional state."

[1864] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1865] Step 1:

[1866] The device detects an incoming call and activates the call recording function. When the call starts, the device automatically starts recording the conversation as digital audio data. The input is the incoming call signal, and the output is the recorded audio data.

[1867] Step 2:

[1868] When the call ends, the device compresses the recorded voice data and sends it to the server. The device transmits the data using a secure protocol over the Internet, maintaining the privacy and security of the data. The input is the recorded voice data, and the output is a data packet containing the compressed voice data.

[1869] Step 3:

[1870] The server passes the received voice data to a speech recognition engine, which converts the voice into text. The speech recognition engine (for example, Google Speech Recognition API) analyzes the voice data and generates corresponding text data. The input is compressed voice data, and the output is text data.

[1871] Step 4:

[1872] The server inputs the generated text data into a natural language processing (NLP) module, which analyzes the text to detect specific keywords and phrases. The NLP module searches for fraud-related keywords (e.g., "transfer" or "prepaid card") within the text and, if detected, proceeds to the next step. The input is the text data, and the output is the detected keyword information.

[1873] Step 5:

[1874] The server inputs the voice data into an emotion recognition engine to analyze the user's emotional state. The emotion recognition engine (e.g., the emotion analysis model in the transformers library) analyzes voice features such as tone, pitch, and intensity to identify the user's emotion (e.g., anxiety, fear). The input is the voice data, and the output is the emotion recognition result.

[1875] Step 6:

[1876] The server integrates the detected keywords and emotion recognition results to comprehensively assess the likelihood of fraud. Based on this result, if there is a high suspicion of fraud, it prepares to send an alert to the user and their family. The input is the keyword detection information and emotion recognition results, and the output is a notification aler...

Claims

1. A means for recording audio; a means for converting recorded speech into text; means for analyzing the generated text to detect specific keywords; means for sending notifications based on the detected keywords; means for storing the detection results and the audio data; A system including:

2. The system of claim 1 further comprising means for sending the notification to a user.

3. The system of claim 1 , further comprising means for sharing the analysis results and audio data with law enforcement agencies.

Images