System

Abstract

A system is provided.SOLUTION: A system installed on a communication device, comprising: means for obtaining user account information; server means for analyzing a user's voice to generate a voice model; means for converting call content to text in real-time; means for analyzing the converted text to determine a suspicion of fraud; means for sending suspicious call content to a server; server means for generating appropriate response text and creating a voice response if a suspicion of fraud is determined; and means for sending the generated voice response to the other party.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] With the current situation of fraud targeting the elderly increasing, there is a need for a method to detect fraud in real time and respond immediately. Furthermore, conventional methods require specialized knowledge to identify fraud and are difficult to respond accurately and quickly, making it difficult for elderly people to respond to fraud themselves. Furthermore, there is a lack of systems that can cover various communication methods, including not only phone calls but also messaging applications and social media, so a comprehensive solution is needed. [Means for solving the problem]

[0005] The present invention provides a means for obtaining a user's account information through an application installed on a communication device. This application works in conjunction with a server that analyzes the user's voice and generates a voice model, converting the contents of the call into text in real time. It also builds a system that analyzes the converted text to determine whether there is suspicion of fraud. If there is a high suspicion of fraud, the relevant call contents are sent to a server, where further detailed analysis is performed and appropriate response text and voice responses are generated. This allows fraud to be prevented by immediately sending an appropriate response to the call recipient. In addition, the system also has the ability to monitor text messages in messaging applications and social networking applications in real time and analyze suspected fraud. This enables fraud prevention not only for calls but also for a variety of communication methods.

[0006] "Communication device" refers to an electronic device capable of making voice calls and sending text messages, such as a phone, smartphone, tablet, or computer.

[0007] "User account information" refers to a set of data including personal information, authentication information, and setting information provided by a user of a communication device.

[0008] "Voice Model" refers to a data set containing individual voice features generated by analyzing a user's voice.

[0009] "Server" refers to a computer system that is remotely accessible and has the computing resources to manage and analyze user data and run generative AI.

[0010] "Means for converting call content into text in real time" refers to voice recognition technology or software for converting voice data during a call into text information in real time.

[0011] "Means for determining suspected fraud" refers to algorithms or rules-based systems that analyze the text of calls and messages to determine whether they are likely to be fraudulent.

[0012] "Means for generating appropriate response text" refers to a function that uses generative AI to create the most appropriate response content when fraud is suspected.

[0013] The "means for generating a voice response" refers to speech synthesis technology for converting the generated response text into natural-sounding speech.

[0014] "Messaging application" refers to software for sending and receiving text messages, such as SMS, chat, and email.

[0015] An "SNS application" is software that provides social networking services, and refers to a platform for communication and information sharing between users. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7]FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] This invention is a system that detects and automatically responds to fraud targeting elderly people in real time through an application installed on a communication device. The system includes server means for acquiring user account information and analyzing the user's voice to generate a voice model. It also includes means for converting call content into text in real time and analyzing the text to determine whether it is suspected of fraud. Suspicious call content is sent to the server, where it is analyzed in detail, and appropriate response text and voice responses are generated and sent to the terminal. This allows users to prevent fraud before it occurs. The system also has the ability to monitor text messages in messaging applications and social networking applications in real time and analyze suspected fraud.

[0038] Program processing overview and explanation

[0039] 1. User interface operation

[0040] First, the user installs and launches the app on their smartphone. They enter their user account information and provide data for identity verification. At this time, the user also records their voice to register. For example, elderly person A installs the app, enters account information such as their name and phone number, and records their voice.

[0041] 2. Server sends initial configuration data

[0042] The server receives the account information and voice data sent by the user, analyzes the user's voice, and generates a voice model. It then sends the initial setup data to the device. For example, the server analyzes person A's voice, generates a voice model, and sends that data to the smartphone.

[0043] 3. Monitor calls and messages

[0044] The device constantly monitors calls and messages in real time. When an incoming call is received, the content of the call is converted into text using voice recognition technology and sent to the analysis engine. For example, if a fraudster calls Person A, the device immediately converts the content of the call into text and begins analysis.

[0045] 4. Sending suspicious verdicts

[0046] The device analyzes the text of the call and detects any elements that may be fraudulent. If fraud is suspected, the device sends the call content to the server. For example, the device may detect phrases that are highly likely to be fraudulent from the call content and send the content to the server.

[0047] 5. Detailed analysis by the server

[0048] The server further analyzes the content of the call sent from the terminal, and if it determines that there is a high suspicion of fraud, it generates an appropriate response text. It uses that text to create a voice response and sends it to the terminal. For example, if the server analyzes the content of the call and determines that it contains a fraud keyword such as "bank transfer," it generates an appropriate response and sends it to the terminal.

[0049] 6. Speech generation and response

[0050] The device uses the received response text to generate a voice response to the call or message, allowing for effective countermeasures against fraudsters. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the device will convey this to the fraudster.

[0051] This series of processes allows users to quickly and effectively deal with even complex and sophisticated fraud methods.In addition, the system can monitor not only the content of phone calls but also text messages on messaging and social networking applications in real time, providing similar countermeasures.

[0052] The processing flow will be explained below.

[0053] Step 1: The user installs the app on their smartphone and launches it. After launching, they enter their user account information, provide identity verification data, and record their voice to register.

[0054] Step 2: The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice to generate an individual voice model. It then sends the generated voice model and initial setting data to the device.

[0055] Step 3: The device stores the initial configuration data received from the server and prepares the app for operation. The voice model and various settings are activated on the device, and the app goes into standby mode.

[0056] Step 4: When the user receives a call, the device detects the incoming call and converts the call content into text in real time using voice recognition technology. At the same time, text messages received via messaging and social networking applications are also monitored.

[0057] Step 5: The device sends the text of the call and messages to an analysis engine, which detects suspicious keywords and phrases. If fraud is suspected, the analysis results and the call content are sent to a server.

[0058] Step 6: The server performs a detailed analysis based on the call content and analysis results received from the device. If it determines that fraud is suspected, it uses the generation AI to generate an appropriate response text.

[0059] Step 7: The server creates a voice response using the voice generation model based on the generated response text, and sends the voice data to the terminal.

[0060] Step 8: The device sends the received voice response to the other party and responds to the fraudster with the generated voice, thereby preventing fraud before it occurs.

[0061] Step 9: After the call ends, the user can check the call history and analysis results within the app, allowing the user to verify whether the call they received was secure.

[0062] Example 1

[0063] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0064] Fraud targeting the elderly is becoming more sophisticated every year, and preventing such damage requires real-time monitoring of call content and rapid response. However, existing systems lack the ability to perform real-time analysis during calls or to promptly notify users, making it difficult to completely prevent fraud. The purpose of this invention is to solve these problems and provide a system that allows elderly people to make calls and send messages with peace of mind.

[0065] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0066] In this invention, the server includes a means for analyzing the user's voice to generate a voice model, a means for analyzing the converted text to determine whether fraud is suspected, and a means for generating an appropriate response text and creating a voice response if fraud is suspected. This allows the server to monitor call content in real time, and if suspicious call content is detected, immediately notify the user of the content and take appropriate measures. It also has a function for notifying the user of suspected fraud when the call ends, allowing elderly people to be quickly protected from the risk of fraud. It is also possible to monitor text messages from messaging applications and social networking service applications in real time and issue a warning if fraud is suspected.

[0067] "Communication device" is a general term for electronic devices that can make calls and send messages, and specifically includes smartphones and tablets.

[0068] "Means for obtaining user account information" refers to the function for collecting personal information entered by the user through the application.

[0069] The "server means for generating a voice model" is a server that has the function of analyzing voice characteristics based on the user's voice data and creating an individual voice profile.

[0070] "Means for converting to text" refers to speech recognition technology that converts voice data into text data in real time.

[0071] "Means for determining suspected fraud" refers to algorithms or engines that analyze text data and determine whether it is likely to be fraudulent.

[0072] "Means for transmitting to a server" refers to a communication function for transmitting data from a terminal to a server.

[0073] The "server means for generating response text and creating a voice response" is a server that has the functionality to generate response text and convert it into voice as an appropriate countermeasure in cases where there is a possibility of fraud.

[0074] "Means for transmitting to the other party" refers to a communication function for transmitting the generated voice response to the other party.

[0075] "Means for notifying" refers to a feature that notifies the user of suspected fraud when the call ends.

[0076] "Warning measures" refers to a function that warns users when a call or message suspected of being fraudulent is detected.

[0077] "Messaging applications and social networking service applications" is a general term for software for sending and receiving text messages in real time, and examples include chat apps and SNS.

[0078] "Converted text" refers to content that has been converted from speech to text using speech recognition technology.

[0079] The present invention is a system for detecting fraud targeting elderly people in real time and automatically dealing with it via an application installed on a communication device. Detailed embodiments of this system are described below.

[0080] First, the user installs a dedicated application on a communication device such as a smartphone. After installation, the user launches the application and enters user account information (such as name, phone number, and email address). Voice data is also recorded and sent to the server along with the account information. This voice data is used to verify the user's identity and to generate a voice model, as described below.

[0081] The server receives the account information and voice data sent by the user. This receiving process is performed using cloud services such as Microsoft Azure and Amazon Web Services (AWS) to ensure high security. The server then analyzes the voice data using voice analysis technologies such as Google Cloud Speech-to-Text and Amazon Polly to generate a voice model specific to the user. This generated voice model and the necessary initial configuration data are then sent to the user's device, ready for real-time voice recognition and monitoring.

[0082] The device then constantly monitors calls and messaging in real time. When a user initiates a call, the device converts the call content into text in real time using voice recognition technology (such as the Google Cloud Speech-to-Text API). This allows the call content to be captured in text format and sent to the analysis engine.

[0083] An analysis engine built into the device (for example, natural language processing technology using Python's NLTK library) analyzes the text of the call and determines whether there is any suspicion of fraud. If it is determined to be suspicious, the call is immediately sent to the server.

[0084] The server then performs further analysis to determine whether the transaction is fraudulent, using a generative AI model (e.g., OpenAI's GPT-4). If fraud is determined to be highly suspected, the server generates an appropriate response text and converts it into a voice response using speech generation technology such as the Google Text-to-Speech API.

[0085] The generated voice response is sent to the device, which then immediately plays it back to the scammer. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the scammer can be informed immediately.

[0086] Furthermore, when the call ends, the device will notify the user that a fraud attempt was suspected. The notification will include a message such as, "There was a suspected fraudulent call. Would you like to check the details?" This notification allows elderly users to quickly understand the risk of fraud and take necessary measures.

[0087] The system not only monitors the content of phone calls but also text messages in messaging apps and social networking applications (SNS) in real time, and performs similar analysis and countermeasures. Similarly, if a suspicious message is detected, it will issue a warning to the user and prompt them to take necessary measures.

[0088] Examples of concrete examples and prompts

[0089] For example, if user A installs the app, registers his account information and voice recording, and then receives a call from a fraudster saying, "Please transfer XXX million yen by today," the contents of the call are converted into text and analyzed in real time. If any suspicious content is detected, the server immediately generates an appropriate countermeasure and sends it to A's device. The device then automatically plays a voice message saying, "This call is suspected to be fraudulent," warning the fraudster.

[0090] Example prompt sentence:

[0091] Please provide specific examples of voice responses that will be triggered when a user receives a call and it is determined to be fraudulent.

[0092] In this way, the system reduces the risk of seniors being scammed in real time, helping them to communicate with peace of mind in their daily lives.

[0093] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0094] Step 1:

[0095] The user installs and launches a dedicated application on their smartphone. The user enters account information such as their name, phone number, and email address, and records voice data. The recorded voice data and account information are sent to the server by the application. The input here is the user's personal information and voice data, and the output is the transmission of information to the server.

[0096] Step 2:

[0097] The server receives the account information and voice data sent by the user. The server uses voice analysis technologies such as Google Cloud Speech-to-Text and Amazon Polly to analyze the voice data and generate a voice model. This voice model is necessary for the subsequent speech recognition process. The input is the user's voice data, and the output is the generated voice model.

[0098] Step 3:

[0099] The server sends the generated voice model and initial setting data to the user's device. The device receives this and prepares to start monitoring calls and messaging. The input is the voice model and initial setting data, and the output is sending the setting data to the device.

[0100] Step 4:

[0101] When a user initiates a call, the device converts the call content into text in real time using speech recognition technology. As a specific example, we use the Google Cloud Speech-to-Text API. The input is the call audio, and the output is the text of the call.

[0102] Step 5:

[0103] The device sends the text of the call to an analysis engine (for example, Python's NLTK library) to determine whether or not there is suspicion of fraud. The input is the text of the call, and the output is the judgment result on whether or not there is suspicion of fraud.

[0104] Step 6:

[0105] If the terminal determines that fraud is suspected, it sends the contents of the call to the server. The input is text data that is suspected of fraud, and the output is data transmission to the server.

[0106] Step 7:

[0107] If the server performs a detailed analysis and determines that there is a high probability of fraud, it generates a response text and creates a voice response. An appropriate response is generated using a generative AI model (e.g., OpenAI's GPT-4) and converted into voice using the Google Text-to-Speech API. The input is the text data suspected of fraud, and the output is the generated voice response.

[0108] Step 8:

[0109] The server sends the generated voice response to the terminal, where the input is the generated voice response and the output is the voice response sent to the terminal.

[0110] Step 9:

[0111] The terminal plays the received voice response and immediately responds to the fraudster. It also notifies the user of the suspected fraud when the call ends. The input is the received voice response data and the call end event, and the output is the voice playback and a notification to the user.

[0112] This series of steps enables the system to detect fraud targeting the elderly in real time and take prompt and appropriate action. The specific actions taken at each processing step and the details of their inputs and outputs effectively protect users from the risk of fraud and enable them to communicate with others in a safe and secure manner.

[0113] (Application example 1)

[0114] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0115] The number of sophisticated frauds targeting the elderly, such as bank transfer scams, is on the rise, making it difficult to quickly and effectively detect and prevent them using conventional methods. Furthermore, frauds are often carried out via voice calls, text messages, and social media, and the lack of a system for monitoring and dealing with these in real time is a problem. Furthermore, when fraud is suspected, it is necessary to quickly issue a warning to family members and related parties.

[0116] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0117] In this invention, the server includes means for acquiring user account information, means for analyzing the user's voice to generate a voice model, means for converting the call content to text in real time, means for analyzing the converted text to determine whether fraud is suspected, means for transmitting the suspicious call content to the server, means for generating an appropriate response text and creating a voice response if fraud is suspected, means for transmitting the generated voice response to the caller, means for issuing a warning notification to family members or related parties if fraud is suspected, and means for monitoring text messages in messaging applications and SNS applications in real time and analyzing whether fraud is suspected. This makes it possible to prevent elderly people from becoming victims of fraud and to quickly and effectively detect and deal with fraudulent activities.

[0118] A "communication device" is an electronic device such as a mobile phone or smartphone that is capable of digital communication.

[0119] "User account information" is data including the user's personal information and authentication information.

[0120] A "voice model" is a database or algorithm that features a user's voice.

[0121] "Server means" refers to a computer and its software set up to process specific data.

[0122] "Means for converting call content into text in real time" refers to speech recognition technology for converting voice into text data.

[0123] The "means for analyzing the converted text to determine suspected fraud" is an algorithm that analyzes the character data to detect fraudulent patterns.

[0124] The "means for transmitting suspicious call content to a server" is a communication technology for uploading suspicious data to a server based on the analysis results.

[0125] The "server means for generating appropriate response text and creating a voice response" is a technology for generating text and converting it into voice so that the server can take appropriate measures against suspicious call content.

[0126] The "means for transmitting the generated voice response to the other party" is a communication technique for transmitting the generated voice data to the other party during the call.

[0127] "Means for sending warning notices to family or related parties" refers to communication technologies for sending warnings to family or related parties in the event of suspected fraud.

[0128] The "means for monitoring text messages on messaging applications and social networking applications in real time" refers to technology for monitoring instant messages and text messages on social networks in real time.

[0129] "Suspected fraud analysis methods" are algorithms that analyze text messages and phone calls to detect fraudulent patterns.

[0130] In this invention, the system for detecting and automatically dealing with fraudulent acts targeting the elderly is implemented using the following hardware and software.

[0131] Hardware Configuration

[0132] Communication devices: Using digital communication devices such as smartphones, smart glasses, head-mounted displays, and domestic robots.

[0133] Server: The server consists of a computer and its software for data processing and analysis.

[0134] Software Configuration

[0135] Speech recognition software: Uses the speech_recognition library to convert speech to text.

[0136] Text analysis software: Uses the TextBlob library to analyze text data to detect suspected fraud.

[0137] Communication Method: Alert notifications are sent via email using the smtplib library.

[0138] Processing flow

[0139] 1. Obtain user account information and generate a speech model:

[0140] A user installs the application on their communication device, enters their account information, records and saves their voice, and the server analyzes the voice and generates a voice model.

[0141] 2. Real-time call monitoring and analysis:

[0142] The communications device constantly monitors calls in real time and converts them into text using speech recognition software, which is then sent to a server where text analysis software determines whether the call is suspected of fraud.

[0143] 3. Dealing with suspicious calls:

[0144] If fraud is suspected based on the analysis results, the system generates an appropriate response text and creates a voice response, which is then sent to the caller, while simultaneously activating communication methods to send warning notifications to households and relevant parties.

[0145] 4. Messaging and Social Media Application Monitoring:

[0146] The communication device monitors text messages in messaging and social networking applications in real time, analyzes the text to detect potential fraud, and similarly sends the data to a server that generates an appropriate response.

[0147] Specific examples

[0148] For example, suppose an elderly person named Mr. A has a smartphone and a fraudster calls him. In this case, the smartphone monitors the contents of the call in real time, converts the voice to text, and if the text contains a fraud keyword such as "transfer." The system immediately sends the content to the server, which analyzes it in detail and automatically generates a response such as "This call is suspected to be fraudulent," warning Mr. A. A warning notification email is also sent to Mr. A's family and associates.

[0149] Prompt Sentence Examples

[0150] For generative AI models, an example prompt might be something like:

[0151] Please create a Python program that recognizes the voice recordings made by users and determines whether the recordings contain keywords that may be fraudulent, such as "transfer" or "remittance." If a keyword is detected, please include a function that notifies the relevant parties by email.

[0152] In this way, the system of the present invention can protect seniors from fraud and take action quickly and effectively.

[0153] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0154] Step 1:

[0155] A user installs an application on a communication device, enters account information, and records voice. This input includes name, phone number, and voice data. This voice data is used to generate a voice model that reflects the user's characteristics. The server analyzes the voice data, generates a personalized voice model, and sends the initialization data to the device. The output is the user's voice model and the initialization data.

[0156] Step 2:

[0157] The communication device constantly monitors the contents of calls in real time. When an incoming call is received, the device collects the contents of the call and converts them into text using voice recognition technology. The input is the voice data collected in real time, and the output is the converted text data.

[0158] Step 3:

[0159] The terminal sends the converted text data to an analysis engine to determine whether it is suspected of fraud. This analysis involves checking whether it contains specific keywords or phrases (e.g., "transfer" or "remittance"). The input is the text data, and the output is a determination of whether it is suspected of fraud.

[0160] Step 4:

[0161] If fraud is suspected, the device sends the text data to a server for further analysis. The server then further analyzes the text data and generates an appropriate response text, which includes an appropriate response to the fraudster. The input is the suspicious text data, and the output is the appropriate response text.

[0162] Step 5:

[0163] The server converts the generated response text into voice data using speech conversion technology and sends this voice data to the terminal. The terminal uses this voice data to immediately respond to the other party. The input is the response text data, and the output is the generated voice data.

[0164] Step 6:

[0165] At the same time, if fraud is suspected, the device will send a warning notification to family members or related parties. The warning notification is sent by means of email or other means and contains details of the suspected fraudulent call. The input is the fraud judgment result, and the output is a warning notification email to family members or related parties.

[0166] Step 7:

[0167] The communication device monitors text messages from messaging and social networking applications in real time and analyzes the text data. If a suspected fraudulent message is detected, it is sent to a server for further analysis. The input is the text message data collected in real time, and the output is a judgment result of whether the message is suspected to be fraudulent and an appropriate response.

[0168] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0169] This invention adds emotion recognition functionality to a system that detects and automatically responds to fraud targeting the elderly in real time through an application installed on a communication device. The invention includes server means for acquiring user account information and analyzing the user's voice to generate a voice model. It also includes means for converting call content into text in real time and analyzing the text to determine whether it is suspected of fraud. Suspicious call content is sent to the server, where it is analyzed in detail, and appropriate response text and voice responses are generated and sent to the terminal. This allows users to prevent fraud before it occurs. The system also has the ability to monitor text messages in messaging applications and social networking applications in real time and analyze suspected fraud.

[0170] Furthermore, this system incorporates an emotion engine that recognizes emotions from the user's voice and text messages during a call. Based on the user's emotions recognized by the emotion engine, the server adjusts the content of the response text. If the user's emotions are negative, such as stress or anxiety, the system has a means to automatically end the call.

[0171] Program processing overview and explanation

[0172] 1. User interface operation

[0173] First, the user installs and launches the app on their smartphone. They enter their user account information and provide data for identity verification. At this time, the user also records their voice to register. For example, elderly person A installs the app, enters account information such as their name and phone number, and records their voice.

[0174] 2. Server sends initial configuration data

[0175] The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice to generate an individual voice model. The generated voice model and initial setting data are sent to the device. For example, the server analyzes person A's voice to generate a voice model and sends that data to the smartphone.

[0176] 3. Monitor calls and messages

[0177] The device constantly monitors calls and messages in real time. When an incoming call is received, the call contents are converted into text using voice recognition technology. At the same time, text messages received via messaging applications and social networking applications are also monitored. For example, if a fraudster calls Person A, the device immediately converts the call contents into text and begins analyzing it.

[0178] 4. Analysis of call content and emotions

[0179] The device sends the text of the call and messages to an analysis engine, which detects suspicious keywords and phrases. It also uses an emotion engine to analyze the user's emotions. If it determines that fraud is suspected, it sends the analysis results and the call content to the server. For example, if the device detects phrases from the call content that are highly suspicious of fraud and determines that the user is feeling anxious, it sends the content to the server.

[0180] 5. Detailed analysis and response generation by the server

[0181] The server performs a detailed analysis based on the call content received from the device and the analysis results. If it determines that fraud is suspected using generative AI, it generates an appropriate response text. If the user's emotions are negative, it adjusts the content of the response text to match their emotions. A voice generation model is used to create a voice response based on the generated response text, and the voice data is sent to the device. For example, if the server analyzes the content of the call and determines that it contains fraud keywords such as "bank transfer," and the user is feeling anxious, it will generate a response including expressions such as "stay calm" and send it to the device.

[0182] 6. Speech generation and response

[0183] The device uses the received response text to generate a voice response to the call or message in question, instantly responding to the call or message. This response allows for effective countermeasures against scammers. If the user is extremely stressed, the call will be automatically terminated. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the device will convey this to the scammer. At the same time, if the user is under high stress, the call will be disconnected.

[0184] This series of processes allows users to quickly and effectively deal with even complex and sophisticated fraud methods. Furthermore, the system can monitor not only phone call content but also text messages on messaging and social networking applications in real time, providing similar countermeasures. By combining it with an emotion engine, the aim is to reduce the psychological burden on users and achieve safer and more secure communication.

[0185] The processing flow will be explained below.

[0186] Step 1: The user installs and launches the app on their smartphone. The user enters their account information and provides data for identity verification. At this time, the user also records their voice to register. For example, an elderly person named Mr. A installs the app, enters account information such as his name and phone number, and records his voice.

[0187] Step 2: The server receives the account information and voice data sent by the user and stores them in a database. Next, it analyzes the user's voice to generate an individual voice model. The generated voice model and initial setting data are sent to the device. For example, the server analyzes person A's voice to generate a voice model and sends that data to the smartphone.

[0188] Step 3: The device stores the initial setup data received from the server and prepares the app for operation. The voice model and various settings are activated on the device, and the app goes into standby mode. For example, the device receives the initial setup data and enables the analysis engine and speech generation engine.

[0189] Step 4: When the user receives a call, the device detects the incoming call and converts the call content into text using voice recognition technology. At the same time, text messages received via messaging and social networking applications are also monitored. For example, if a fraudster calls Person A, the device will immediately convert the call content into text and begin analyzing it.

[0190] Step 5: The device sends the converted call content and messages to an analysis engine, which detects suspicious keywords and phrases. It also uses an emotion engine to analyze the user's emotions. If it determines that fraud is suspected, it sends the analysis results and the call content to the server. For example, if the device detects phrases from the call content that are highly suspicious of fraud and determines that the user is feeling anxious, it sends the content to the server.

[0191] Step 6: The server performs a detailed analysis based on the call content received from the device and the analysis results. If it determines using generative AI that fraud is suspected, it generates an appropriate response text. If the user's emotions are negative, the content of the response text is adjusted to match the emotion. For example, if the server analyzes the call content and determines that it contains fraud keywords such as "transfer," and the user is feeling anxious, it will generate a response including expressions such as "remain calm" and send it to the device.

[0192] Step 7: The server creates a voice response using the speech generation model based on the generated response text and sends the voice data to the device. For example, the server generates a response such as "This call is suspected to be fraudulent. We will report this to the police" and sends it to the device as voice data.

[0193] Step 8: The device sends the received voice response to the other party and responds to the fraudster with a generated voice. This response can prevent fraud before it occurs. If the user is extremely stressed, the call can be automatically terminated. For example, if the device tells the fraudster in a generated voice, "This call is suspected to be fraud. We will report this to the police," and at the same time detects that the user is stressed, the call will be disconnected.

[0194] Step 9: After the call, the user can check the call history and analysis results within the app. This allows the user to confirm whether the call they received was safe. For example, if Person A checks the app after the call ends and learns from the analysis results that there was a suspicion of fraud.

[0195] Example 2

[0196] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0197] Fraud targeting the elderly is becoming more sophisticated every year, and is often difficult to prevent using conventional methods. Furthermore, real-time responses are required when a user is scammed, but existing technologies make it difficult to provide immediate responses. Furthermore, elderly users, who are more likely to fall victim to fraud, often experience anxiety and stress, and methods to alleviate this stress are also needed. Existing technologies lack systems that provide automated responses that take the user's emotional state into account, so there is an urgent need to develop a system that can reduce the psychological burden and prevent fraud quickly and effectively.

[0198] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0199] In this invention, the server includes means for acquiring user account information, means for analyzing the user's voice to generate a voice model, means for converting call content into text in real time, means for analyzing the converted text to determine whether a call is suspected of fraud, means for transmitting suspicious call content to the server, means for generating appropriate response text and creating a voice response using a generative AI model if a fraud is determined to be suspected, emotion recognition means for analyzing the user's emotions in real time, and means for automatically terminating the call based on the user's negative emotions. This makes it possible to detect calls or messages that are suspected of being fraudulent in real time and respond immediately. Furthermore, automatically terminating calls taking into account the user's emotional state significantly reduces psychological burden.

[0200] The "means for obtaining user account information" is a system component for obtaining personal information such as a user's name, phone number, email address, etc. through an application installed on a communication device.

[0201] The "server means for analyzing the user's voice and generating a voice model" is a system component on the server side that analyzes the voice data provided by the user and generates a voice model based on the voice characteristics.

[0202] "Means for converting call content into text in real time" refers to technology for converting voice data of incoming and outgoing calls into text data in real time.

[0203] The "means for analyzing the converted text to determine suspicion of fraud" is an analytical engine for analyzing the text content of the call and determining whether there is suspicion of fraud.

[0204] The "means for sending suspicious call content to a server" is a system component for sending the call content to a server if it is determined to be suspicious of fraud.

[0205] A "server means using a generative AI model" is a server-side system component that utilizes generative AI technology to generate appropriate response text and create a voice response based on that text.

[0206] "Emotion recognition means for analyzing user emotions in real time" is a technology for identifying and analyzing emotions in real time from a user's voice or text message.

[0207] The "means for automatically terminating a call based on a user's negative emotions" is a system component that automatically terminates a call when the user has negative emotions such as stress or anxiety.

[0208] This invention adds emotion recognition functionality to a system that detects and automatically responds to fraud targeting the elderly in real time. The invention includes a server means that acquires user account information through an application installed on a communication device, analyzes the user's voice, and generates a voice model.

[0209] Hardware and software used

[0210] Speech recognition technology: Uses the Google Cloud Speech-to-Text API to convert calls into text in real time.

[0211] Emotion Engine: Analyzes user emotions in real time using the Microsoft Azure Emotion API.

[0212] Database: Amazon RDS is used to store and manage user account information and voice models.

[0213] Generative AI: Uses OpenAI's GPT-3 to generate appropriate response text when suspected fraud is detected.

[0214] Call Management System: Uses Asterisk to capture real-time call data and perform text transcription.

[0215] Process Overview

[0216] First, the user installs and launches the app on their smartphone. They enter their user account information and provide data for identity verification. At this time, the user also records their voice to register. For example, elderly person A installs the app, enters account information such as their name and phone number, and records their voice.

[0217] The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice to generate an individual voice model. The generated voice model and initial setting data are sent to the device. For example, the server analyzes person A's voice to generate a voice model and sends that data to the smartphone.

[0218] The device constantly monitors calls and messages in real time. When an incoming call is received, the call contents are converted into text using voice recognition technology. At the same time, text messages received via messaging applications and social networking applications are also monitored. For example, if a fraudster calls Person A, the device immediately converts the call contents into text and begins analyzing it.

[0219] The device sends the text of the call and messages to an analysis engine, which detects suspicious keywords and phrases. It also uses an emotion engine to analyze the user's emotions. If it determines that fraud is suspected, it sends the analysis results and the call content to the server. For example, if the device detects phrases from the call content that are highly suspicious of fraud and determines that the user is feeling anxious, it sends the content to the server.

[0220] The server performs a detailed analysis based on the call content received from the device and the analysis results. If it determines that fraud is suspected using generative AI, it generates an appropriate response text. If the user's emotions are negative, it adjusts the content of the response text to match their emotions. A voice generation model is used to create a voice response based on the generated response text, and the voice data is sent to the device. For example, if the server analyzes the content of the call and determines that it contains fraud keywords such as "bank transfer" and the user is feeling anxious, it will generate a response including expressions such as "stay calm" and send it to the device.

[0221] The device uses the received response text to generate a voice response to the call or message in question, instantly responding to the call or message. This response allows for effective countermeasures against scammers. If the user is extremely stressed, the call will be automatically terminated. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the device will convey this to the scammer. At the same time, if the user is under high stress, the call will be disconnected.

[0222] Specific prompt sentence examples and output examples

[0223] Prompt statement:

[0224] User says: "This is from your bank. We're having trouble with your account..."

[0225] Analysis results:

[0226] Fraud suspicion level: High

[0227] Sentiment analysis: Determines that the user's emotion is "anxiety"

[0228] Generates the following response:

[0229] This call is suspected to be fraudulent and I will be reporting the details to the police.

[0230] Example output (response after speech generation):

[0231] This call is suspected to be a scam. I will report the details to the police. Please remain calm.

[0232] This invention allows users to prevent fraud and alleviate psychological stress through emotion recognition. It provides a system that monitors calls and analyzes emotions in real time, allowing for quick and effective response.

[0233] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0234] Step 1:

[0235] A user installs and launches an app on their smartphone, which includes downloading and installing it from the app store and then tapping the app's icon to launch it.

[0236] Input: User terminal operation.

[0237] Output: The app is launched.

[0238] Step 2:

[0239] Users enter their user account information within the app and provide data to verify their identity, such as their name, phone number, and email address, and record and register their own voice to create a voice model.

[0240] Input: Name, phone number, email address, recorded voice data.

[0241] Output: Completed user account information and audio data.

[0242] Step 3:

[0243] The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice using speech recognition technology (Google Cloud Speech-to-Text API) to generate a voice model. The generated voice model and initial setting data are sent to the device.

[0244] Input: User account information, recorded voice data.

[0245] Output: User information stored in the database. Generated speech model and initial configuration data.

[0246] Step 4:

[0247] The device goes into constant monitoring mode and monitors calls and messages in real time. When an incoming call is received, the conversation is captured and converted into text using the Google Cloud Speech-to-Text API. Similarly, text messages received via messaging and social networking applications are also monitored.

[0248] Input: Calls and text messages.

[0249] Output: Transcript of calls and monitoring records.

[0250] Step 5:

[0251] The device sends the transcribed conversations and messages to an analytics engine that detects suspicious keywords and phrases and analyzes the user's emotions using the Microsoft Azure Emotion API.

[0252] Input: Texted calls and messages.

[0253] Output: Detected suspicious keywords and phrases, analyzed sentiment data.

[0254] Step 6:

[0255] The device sends the analysis results to the server. The server performs a detailed analysis based on the received analysis results and the content of the call. If it determines that fraud is suspected using generative AI (GPT-3), it generates an appropriate response text. If the user's sentiment is judged to be negative, the content of the response text is adjusted.

[0256] Input: Analysis results, emotion data.

[0257] Output: Detailed analysis results, generated response text.

[0258] Step 7:

[0259] The server creates a voice response using a speech generation model based on the generated response text, and sends the generated voice data to the terminal.

[0260] Input: The response text.

[0261] Output: The generated audio data.

[0262] Step 8:

[0263] The device responds to calls and messages in real time using the voice data it receives, and automatically ends calls if the user is feeling extremely stressed.

[0264] Input: The generated audio data.

[0265] Output: Answered voice message, call ended.

[0266] This process effectively protects users from fraud targeting the elderly, and real-time monitoring and sentiment analysis enable swift and appropriate responses.

[0267] (Application example 2)

[0268] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0269] Fraud targeting the elderly is increasing year by year, and the methods are becoming more sophisticated and ingenious. In particular, fraud perpetrated via telephone or messaging services has the potential to cause significant damage if not dealt with promptly. Furthermore, victims often experience significant emotional stress and anxiety when they encounter fraud. However, current fraud prevention systems lack the ability to address these emotional needs. Therefore, the present invention aims to provide a more effective and secure fraud prevention system for the elderly by combining fraud detection and emotion recognition.

[0270] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server is installed in the communication device and includes means for acquiring user account information, means for analyzing the user's voice to generate a voice model, and means for converting the content of the call into text in real time. This includes means for generating appropriate response text and voice response while taking the user's emotional state into consideration when fraud is suspected, and means for adjusting the content of the response text if the user's emotions are negative and automatically terminating the call if the user's stress level is high. This not only detects fraudulent acts but also enables fraud prevention measures that reduce the mental burden on elderly people and allow them to use the service with peace of mind.

[0271] "Communication device" refers to a device that transmits and receives information using telecommunications technology, including smartphones and tablets.

[0272] "User account information" refers to data containing a user's personal information and authentication information, such as name, phone number, and password.

[0273] A "voice model" is a computer model generated by analyzing the vocal characteristics of a particular user and used to reproduce the user's voice patterns.

[0274] "Server means" refers to a collection of server devices and server programs for providing specific functions and services.

[0275] "Means for converting to text" refers to the technology or device that converts voice data into text data. A voice recognition engine is an example of this.

[0276] "Means for analyzing text to determine suspected fraud" refers to technologies or programs that analyze converted character data and assess the likelihood of fraud.

[0277] "Emotion engine" refers to algorithms or software that identify a user's emotional state from speech or text.

[0278] "Negative emotions" refers to negative emotional states such as anxiety, stress, anger, and sadness.

[0279] "Means for generating appropriate response text" refers to technology or a program that creates an appropriate message for the user or the other party based on the analysis results.

[0280] "Voice response" refers to generating voice data that reads out text data.

[0281] "Means for automatically terminating a call" refers to a mechanism or technology that forcibly terminates a call when certain conditions are met.

[0282] This invention provides a system for detecting and automatically combating fraud targeting seniors in real time through an application installed on a communication device. The invention includes the following main components:

[0283] User interface operations

[0284] First, the user installs and launches the application on their smartphone. They must enter their account information and provide data for identity verification. At this point, the user registers by recording their voice. For example, an elderly person named Mr. A installs the app, enters account information such as his name and phone number, and records his voice.

[0285] Server sends initial configuration data

[0286] The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice to generate an individual voice model. The generated voice model and initial setting data are sent to the device. For example, the server analyzes person A's voice to generate a voice model and sends that data to the smartphone.

[0287] Call and message monitoring

[0288] The device constantly monitors calls and messages in real time. When an incoming call is received, the contents of the call are converted into text using voice recognition technology. At the same time, text messages received via messaging applications and social networking applications are also monitored. For example, if Person A receives a call from a fraudster, the device immediately converts the contents of the call into text and begins analysis.

[0289] Call content and emotion analysis

[0290] After the call contents are converted into text, they are sent to an analysis engine, which detects suspicious keywords and phrases. The emotion engine also analyzes the user's emotions. If fraud is suspected, the analysis results and the contents of the call are sent to the server. For example, if the device detects phrases from the call contents that are highly suspicious of fraud and determines that the user is feeling anxious, the content is sent to the server.

[0291] Detailed analysis and response generation by the server

[0292] The server performs a detailed analysis based on the call content received from the device and the analysis results. If it determines that fraud is suspected using a generative AI model, it generates an appropriate response text. If the user's emotions are negative, it adjusts the content of the response text to match the emotion. Based on the generated response text, it creates a voice response using a voice generation model, and sends the voice data to the device. For example, if the server analyzes the content of the call and determines that it contains fraud keywords such as "bank transfer," and the user is feeling anxious, it generates a response including the expression "stay calm" and sends it to the device.

[0293] Speech generation and response

[0294] The device uses the received response text to generate a voice response to the call or message in question, instantly responding to the call or message. This response allows for effective countermeasures against scammers. If the user is extremely stressed, the call will be automatically terminated. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the device will convey this to the scammer. At the same time, if the user is under high stress, the call will be disconnected.

[0295] Examples and prompts

[0296] For example, consider the case where elderly person A receives a phone call that is suspected to be a scam. Speech recognition recognizes A's statement that "I need to transfer money," and emotion analysis determines that the stress score is high (e.g., 0.9). The fraud detection model detects the suspicion of fraud from the keyword "transfer," and the response generation model creates an appropriate response such as, "This call may be a scam. Please remain calm and do not respond to the caller's requests. The call will end."

[0297] Example prompt for a generative AI model:

[0298] We provide the call content so you can determine the emotional state: "I need to transfer money."

[0299] This invention will enable elderly people to prevent themselves from falling victim to fraud, reduce their mental burden, and live their daily lives with peace of mind.

[0300] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0301] Step 1:

[0302] User interface operations:

[0303] A user installs and launches an application on a communication device. The user enters account information (such as name and phone number) and records their voice to register. At this time, the communication device collects the entered account information and voice data and sends them to a server. The input data is the user's personal information and voice data, and the output is initial setting data sent to the server.

[0304] Step 2:

[0305] Server sends initialization data:

[0306] The server receives the account information and voice data sent by the user and stores them in a database. The server then analyzes the received voice data to generate a voice model. This voice model reproduces the user's voice characteristics. The generated voice model and initial setting data are sent to the communication device. The data processing required in this process is voice analysis, and the output is a voice model and initial setting data.

[0307] Step 3:

[0308] Call and message monitoring:

[0309] The communication device monitors all calls and messaging application interactions in real time. When an incoming call is received, the call content is converted into text using voice recognition technology. It also monitors text messages from messaging applications and social networking applications. In this monitoring process, the input is voice data and text data, and the output is the converted call content.

[0310] Step 4:

[0311] Call content and sentiment analysis:

[0312] The communication device sends the text of the call and messages to an analysis engine to detect keywords and phrases that may indicate fraud. It also uses an emotion engine to analyze the user's emotional state. If it determines that fraud is suspected, it sends that information to a server. In this step, the input is the text of the call, and the output is the analysis result and the emotional state.

[0313] Step 5:

[0314] Detailed analysis and response generation by the server:

[0315] The server performs a detailed analysis based on the call content and analysis results sent from the communication device. If suspicion of fraud is confirmed, the server uses a generative AI model to generate an appropriate response text. It also adjusts the response content based on the user's emotional state. It creates a voice response using a speech generation model based on the generated response text, and sends the voice data to the communication device. The inputs to this step are the analysis results and the emotional state, and the output is voice response data.

[0316] Step 6:

[0317] Speech generation and response:

[0318] The communication device generates a voice based on the response text received from the server and immediately responds to the caller or message recipient. It also automatically ends the call if the user is extremely stressed. In this process, the input is the voice response data and the output is the end of the call or message.

[0319] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0320] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0321] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0322] [Second embodiment]

[0323] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0324] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0325] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0326] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0327] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0328] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0329] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0330] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0331] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0332] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0333] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0334] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0335] This invention is a system that detects and automatically responds to fraud targeting elderly people in real time through an application installed on a communication device. The system includes server means for acquiring user account information and analyzing the user's voice to generate a voice model. It also includes means for converting call content into text in real time and analyzing the text to determine whether it is suspected of fraud. Suspicious call content is sent to the server, where it is analyzed in detail, and appropriate response text and voice responses are generated and sent to the terminal. This allows users to prevent fraud before it occurs. The system also has the ability to monitor text messages in messaging applications and social networking applications in real time and analyze suspected fraud.

[0336] Program processing overview and explanation

[0337] 1. User interface operation

[0338] First, the user installs and launches the app on their smartphone. They enter their user account information and provide data for identity verification. At this time, the user also records their voice to register. For example, elderly person A installs the app, enters account information such as their name and phone number, and records their voice.

[0339] 2. Server sends initial configuration data

[0340] The server receives the account information and voice data sent by the user, analyzes the user's voice, and generates a voice model. It then sends the initial setup data to the device. For example, the server analyzes person A's voice, generates a voice model, and sends that data to the smartphone.

[0341] 3. Monitor calls and messages

[0342] The device constantly monitors calls and messages in real time. When an incoming call is received, the content of the call is converted into text using voice recognition technology and sent to the analysis engine. For example, if a fraudster calls Person A, the device immediately converts the content of the call into text and begins analysis.

[0343] 4. Sending suspicious verdicts

[0344] The device analyzes the text of the call and detects any elements that may be fraudulent. If fraud is suspected, the device sends the call content to the server. For example, the device may detect phrases that are highly likely to be fraudulent from the call content and send the content to the server.

[0345] 5. Detailed analysis by the server

[0346] The server further analyzes the content of the call sent from the terminal, and if it determines that there is a high suspicion of fraud, it generates an appropriate response text. It uses that text to create a voice response and sends it to the terminal. For example, if the server analyzes the content of the call and determines that it contains a fraud keyword such as "bank transfer," it generates an appropriate response and sends it to the terminal.

[0347] 6. Speech generation and response

[0348] The device uses the received response text to generate a voice response to the call or message, allowing for effective countermeasures against fraudsters. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the device will convey this to the fraudster.

[0349] This series of processes allows users to quickly and effectively deal with even complex and sophisticated fraud methods.In addition, the system can monitor not only the content of phone calls but also text messages on messaging and social networking applications in real time, providing similar countermeasures.

[0350] The processing flow will be explained below.

[0351] Step 1: The user installs the app on their smartphone and launches it. After launching, they enter their user account information, provide identity verification data, and record their voice to register.

[0352] Step 2: The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice to generate an individual voice model. It then sends the generated voice model and initial setting data to the device.

[0353] Step 3: The device stores the initial configuration data received from the server and prepares the app for operation. The voice model and various settings are activated on the device, and the app goes into standby mode.

[0354] Step 4: When the user receives a call, the device detects the incoming call and converts the call content into text in real time using voice recognition technology. At the same time, text messages received via messaging and social networking applications are also monitored.

[0355] Step 5: The device sends the text of the call and messages to an analysis engine, which detects suspicious keywords and phrases. If fraud is suspected, the analysis results and the call content are sent to a server.

[0356] Step 6: The server performs a detailed analysis based on the call content and analysis results received from the device. If it determines that fraud is suspected, it uses the generation AI to generate an appropriate response text.

[0357] Step 7: The server creates a voice response using the voice generation model based on the generated response text, and sends the voice data to the terminal.

[0358] Step 8: The device sends the received voice response to the other party and responds to the fraudster with the generated voice, thereby preventing fraud before it occurs.

[0359] Step 9: After the call ends, the user can check the call history and analysis results within the app, allowing the user to verify whether the call they received was secure.

[0360] Example 1

[0361] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0362] Fraud targeting the elderly is becoming more sophisticated every year, and preventing such damage requires real-time monitoring of call content and rapid response. However, existing systems lack the ability to perform real-time analysis during calls or to promptly notify users, making it difficult to completely prevent fraud. The purpose of this invention is to solve these problems and provide a system that allows elderly people to make calls and send messages with peace of mind.

[0363] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0364] In this invention, the server includes a means for analyzing the user's voice to generate a voice model, a means for analyzing the converted text to determine whether fraud is suspected, and a means for generating an appropriate response text and creating a voice response if fraud is suspected. This allows the server to monitor call content in real time, and if suspicious call content is detected, immediately notify the user of the content and take appropriate measures. It also has a function for notifying the user of suspected fraud when the call ends, allowing elderly people to be quickly protected from the risk of fraud. It is also possible to monitor text messages from messaging applications and social networking service applications in real time and issue a warning if fraud is suspected.

[0365] "Communication device" is a general term for electronic devices that can make calls and send messages, and specifically includes smartphones and tablets.

[0366] "Means for obtaining user account information" refers to the function for collecting personal information entered by the user through the application.

[0367] The "server means for generating a voice model" is a server that has the function of analyzing voice characteristics based on the user's voice data and creating an individual voice profile.

[0368] "Means for converting to text" refers to speech recognition technology that converts voice data into text data in real time.

[0369] "Means for determining suspected fraud" refers to algorithms or engines that analyze text data and determine whether it is likely to be fraudulent.

[0370] "Means for transmitting to a server" refers to a communication function for transmitting data from a terminal to a server.

[0371] The "server means for generating response text and creating a voice response" is a server that has the functionality to generate response text and convert it into voice as an appropriate countermeasure in cases where there is a possibility of fraud.

[0372] "Means for transmitting to the other party" refers to a communication function for transmitting the generated voice response to the other party.

[0373] "Means for notifying" refers to a feature that notifies the user of suspected fraud when the call ends.

[0374] "Warning measures" refers to a function that warns users when a call or message suspected of being fraudulent is detected.

[0375] "Messaging applications and social networking service applications" is a general term for software for sending and receiving text messages in real time, and examples include chat apps and SNS.

[0376] "Converted text" refers to content that has been converted from speech to text using speech recognition technology.

[0377] The present invention is a system for detecting fraud targeting elderly people in real time and automatically dealing with it via an application installed on a communication device. Detailed embodiments of this system are described below.

[0378] First, the user installs a dedicated application on a communication device such as a smartphone. After installation, the user launches the application and enters user account information (such as name, phone number, and email address). Voice data is also recorded and sent to the server along with the account information. This voice data is used to verify the user's identity and to generate a voice model, as described below.

[0379] The server receives the account information and voice data sent by the user. This receiving process is performed using cloud services such as Microsoft Azure and Amazon Web Services (AWS) to ensure high security. The server then analyzes the voice data using voice analysis technologies such as Google Cloud Speech-to-Text and Amazon Polly to generate a voice model specific to the user. This generated voice model and the necessary initial configuration data are then sent to the user's device, ready for real-time voice recognition and monitoring.

[0380] The device then constantly monitors calls and messaging in real time. When a user initiates a call, the device converts the call content into text in real time using voice recognition technology (such as the Google Cloud Speech-to-Text API). This allows the call content to be captured in text format and sent to the analysis engine.

[0381] An analysis engine built into the device (for example, natural language processing technology using Python's NLTK library) analyzes the text of the call and determines whether there is any suspicion of fraud. If it is determined to be suspicious, the call is immediately sent to the server.

[0382] The server then performs further analysis to determine whether the transaction is fraudulent, using a generative AI model (e.g., OpenAI's GPT-4). If fraud is determined to be highly suspected, the server generates an appropriate response text and converts it into a voice response using speech generation technology such as the Google Text-to-Speech API.

[0383] The generated voice response is sent to the device, which then immediately plays it back to the scammer. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the scammer can be informed immediately.

[0384] Furthermore, when the call ends, the device will notify the user that a fraud attempt was suspected. The notification will include a message such as, "There was a suspected fraudulent call. Would you like to check the details?" This notification allows elderly users to quickly understand the risk of fraud and take necessary measures.

[0385] The system not only monitors the content of phone calls but also text messages in messaging apps and social networking applications (SNS) in real time, and performs similar analysis and countermeasures. Similarly, if a suspicious message is detected, it will issue a warning to the user and prompt them to take necessary measures.

[0386] Examples of concrete examples and prompts

[0387] For example, if user A installs the app, registers his account information and voice recording, and then receives a call from a fraudster saying, "Please transfer XXX million yen by today," the contents of the call are converted into text and analyzed in real time. If any suspicious content is detected, the server immediately generates an appropriate countermeasure and sends it to A's device. The device then automatically plays a voice message saying, "This call is suspected to be fraudulent," warning the fraudster.

[0388] Example prompt sentence:

[0389] Please provide specific examples of voice responses that will be triggered when a user receives a call and it is determined to be fraudulent.

[0390] In this way, the system reduces the risk of seniors being scammed in real time, helping them to communicate with peace of mind in their daily lives.

[0391] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0392] Step 1:

[0393] The user installs and launches a dedicated application on their smartphone. The user enters account information such as their name, phone number, and email address, and records voice data. The recorded voice data and account information are sent to the server by the application. The input here is the user's personal information and voice data, and the output is the transmission of information to the server.

[0394] Step 2:

[0395] The server receives the account information and voice data sent by the user. The server uses voice analysis technologies such as Google Cloud Speech-to-Text and Amazon Polly to analyze the voice data and generate a voice model. This voice model is necessary for the subsequent speech recognition process. The input is the user's voice data, and the output is the generated voice model.

[0396] Step 3:

[0397] The server sends the generated voice model and initial setting data to the user's device. The device receives this and prepares to start monitoring calls and messaging. The input is the voice model and initial setting data, and the output is sending the setting data to the device.

[0398] Step 4:

[0399] When a user initiates a call, the device converts the call content into text in real time using speech recognition technology. As a specific example, we use the Google Cloud Speech-to-Text API. The input is the call audio, and the output is the text of the call.

[0400] Step 5:

[0401] The device sends the text of the call to an analysis engine (for example, Python's NLTK library) to determine whether or not there is suspicion of fraud. The input is the text of the call, and the output is the judgment result on whether or not there is suspicion of fraud.

[0402] Step 6:

[0403] If the terminal determines that fraud is suspected, it sends the contents of the call to the server. The input is text data that is suspected of fraud, and the output is data transmission to the server.

[0404] Step 7:

[0405] If the server performs a detailed analysis and determines that there is a high probability of fraud, it generates a response text and creates a voice response. An appropriate response is generated using a generative AI model (e.g., OpenAI's GPT-4) and converted into voice using the Google Text-to-Speech API. The input is the text data suspected of fraud, and the output is the generated voice response.

[0406] Step 8:

[0407] The server sends the generated voice response to the terminal, where the input is the generated voice response and the output is the voice response sent to the terminal.

[0408] Step 9:

[0409] The terminal plays the received voice response and immediately responds to the fraudster. It also notifies the user of the suspected fraud when the call ends. The input is the received voice response data and the call end event, and the output is the voice playback and a notification to the user.

[0410] This series of steps enables the system to detect fraud targeting the elderly in real time and take prompt and appropriate action. The specific actions taken at each processing step and the details of their inputs and outputs effectively protect users from the risk of fraud and enable them to communicate with others in a safe and secure manner.

[0411] (Application example 1)

[0412] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0413] The number of sophisticated frauds targeting the elderly, such as bank transfer scams, is on the rise, making it difficult to quickly and effectively detect and prevent them using conventional methods. Furthermore, frauds are often carried out via voice calls, text messages, and social media, and the lack of a system for monitoring and dealing with these in real time is a problem. Furthermore, when fraud is suspected, it is necessary to quickly issue a warning to family members and related parties.

[0414] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0415] In this invention, the server includes means for acquiring user account information, means for analyzing the user's voice to generate a voice model, means for converting the call content to text in real time, means for analyzing the converted text to determine whether fraud is suspected, means for transmitting the suspicious call content to the server, means for generating an appropriate response text and creating a voice response if fraud is suspected, means for transmitting the generated voice response to the caller, means for issuing a warning notification to family members or related parties if fraud is suspected, and means for monitoring text messages in messaging applications and SNS applications in real time and analyzing whether fraud is suspected. This makes it possible to prevent elderly people from becoming victims of fraud and to quickly and effectively detect and deal with fraudulent activities.

[0416] A "communication device" is an electronic device such as a mobile phone or smartphone that is capable of digital communication.

[0417] "User account information" is data including the user's personal information and authentication information.

[0418] A "voice model" is a database or algorithm that features a user's voice.

[0419] "Server means" refers to a computer and its software set up to process specific data.

[0420] "Means for converting call content into text in real time" refers to speech recognition technology for converting voice into text data.

[0421] The "means for analyzing the converted text to determine suspected fraud" is an algorithm that analyzes the character data to detect fraudulent patterns.

[0422] The "means for transmitting suspicious call content to a server" is a communication technology for uploading suspicious data to a server based on the analysis results.

[0423] The "server means for generating appropriate response text and creating a voice response" is a technology for generating text and converting it into voice so that the server can take appropriate measures against suspicious call content.

[0424] The "means for transmitting the generated voice response to the other party" is a communication technique for transmitting the generated voice data to the other party during the call.

[0425] "Means for sending warning notices to family or related parties" refers to communication technologies for sending warnings to family or related parties in the event of suspected fraud.

[0426] The "means for monitoring text messages on messaging applications and social networking applications in real time" refers to technology for monitoring instant messages and text messages on social networks in real time.

[0427] "Suspected fraud analysis methods" are algorithms that analyze text messages and phone calls to detect fraudulent patterns.

[0428] In this invention, the system for detecting and automatically dealing with fraudulent acts targeting the elderly is implemented using the following hardware and software.

[0429] Hardware Configuration

[0430] Communication devices: Using digital communication devices such as smartphones, smart glasses, head-mounted displays, and domestic robots.

[0431] Server: The server consists of a computer and its software for data processing and analysis.

[0432] Software Configuration

[0433] Speech recognition software: Uses the speech_recognition library to convert speech to text.

[0434] Text analysis software: Uses the TextBlob library to analyze text data to detect suspected fraud.

[0435] Communication Method: Alert notifications are sent via email using the smtplib library.

[0436] Processing flow

[0437] 1. Obtain user account information and generate a speech model:

[0438] A user installs the application on their communication device, enters their account information, records and saves their voice, and the server analyzes the voice and generates a voice model.

[0439] 2. Real-time call monitoring and analysis:

[0440] The communications device constantly monitors calls in real time and converts them into text using speech recognition software, which is then sent to a server where text analysis software determines whether the call is suspected of fraud.

[0441] 3. Dealing with suspicious calls:

[0442] If fraud is suspected based on the analysis results, the system generates an appropriate response text and creates a voice response, which is then sent to the caller, while simultaneously activating communication methods to send warning notifications to households and relevant parties.

[0443] 4. Messaging and Social Media Application Monitoring:

[0444] The communication device monitors text messages in messaging and social networking applications in real time, analyzes the text to detect potential fraud, and similarly sends the data to a server that generates an appropriate response.

[0445] Specific examples

[0446] For example, suppose an elderly person named Mr. A has a smartphone and a fraudster calls him. In this case, the smartphone monitors the contents of the call in real time, converts the voice to text, and if the text contains a fraud keyword such as "transfer." The system immediately sends the content to the server, which analyzes it in detail and automatically generates a response such as "This call is suspected to be fraudulent," warning Mr. A. A warning notification email is also sent to Mr. A's family and associates.

[0447] Prompt Sentence Examples

[0448] For generative AI models, an example prompt might be something like:

[0449] Please create a Python program that recognizes the voice recordings made by users and determines whether the recordings contain keywords that may be fraudulent, such as "transfer" or "remittance." If a keyword is detected, please include a function that notifies the relevant parties by email.

[0450] In this way, the system of the present invention can protect seniors from fraud and take action quickly and effectively.

[0451] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0452] Step 1:

[0453] A user installs an application on a communication device, enters account information, and records voice. This input includes name, phone number, and voice data. This voice data is used to generate a voice model that reflects the user's characteristics. The server analyzes the voice data, generates a personalized voice model, and sends the initialization data to the device. The output is the user's voice model and the initialization data.

[0454] Step 2:

[0455] The communication device constantly monitors the contents of calls in real time. When an incoming call is received, the device collects the contents of the call and converts them into text using voice recognition technology. The input is the voice data collected in real time, and the output is the converted text data.

[0456] Step 3:

[0457] The terminal sends the converted text data to an analysis engine to determine whether it is suspected of fraud. This analysis involves checking whether it contains specific keywords or phrases (e.g., "transfer" or "remittance"). The input is the text data, and the output is a determination of whether it is suspected of fraud.

[0458] Step 4:

[0459] If fraud is suspected, the device sends the text data to a server for further analysis. The server then further analyzes the text data and generates an appropriate response text, which includes an appropriate response to the fraudster. The input is the suspicious text data, and the output is the appropriate response text.

[0460] Step 5:

[0461] The server converts the generated response text into voice data using speech conversion technology and sends this voice data to the terminal. The terminal uses this voice data to immediately respond to the other party. The input is the response text data, and the output is the generated voice data.

[0462] Step 6:

[0463] At the same time, if fraud is suspected, the device will send a warning notification to family members or related parties. The warning notification is sent by means of email or other means and contains details of the suspected fraudulent call. The input is the fraud judgment result, and the output is a warning notification email to family members or related parties.

[0464] Step 7:

[0465] The communication device monitors text messages from messaging and social networking applications in real time and analyzes the text data. If a suspected fraudulent message is detected, it is sent to a server for further analysis. The input is the text message data collected in real time, and the output is a judgment result of whether the message is suspected to be fraudulent and an appropriate response.

[0466] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0467] This invention adds emotion recognition functionality to a system that detects and automatically responds to fraud targeting the elderly in real time through an application installed on a communication device. The invention includes server means for acquiring user account information and analyzing the user's voice to generate a voice model. It also includes means for converting call content into text in real time and analyzing the text to determine whether it is suspected of fraud. Suspicious call content is sent to the server, where it is analyzed in detail, and appropriate response text and voice responses are generated and sent to the terminal. This allows users to prevent fraud before it occurs. The system also has the ability to monitor text messages in messaging applications and social networking applications in real time and analyze suspected fraud.

[0468] Furthermore, this system incorporates an emotion engine that recognizes emotions from the user's voice and text messages during a call. Based on the user's emotions recognized by the emotion engine, the server adjusts the content of the response text. If the user's emotions are negative, such as stress or anxiety, the system has a means to automatically end the call.

[0469] Program processing overview and explanation

[0470] 1. User interface operation

[0471] First, the user installs and launches the app on their smartphone. They enter their user account information and provide data for identity verification. At this time, the user also records their voice to register. For example, elderly person A installs the app, enters account information such as their name and phone number, and records their voice.

[0472] 2. Server sends initial configuration data

[0473] The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice to generate an individual voice model. The generated voice model and initial setting data are sent to the device. For example, the server analyzes person A's voice to generate a voice model and sends that data to the smartphone.

[0474] 3. Monitor calls and messages

[0475] The device constantly monitors calls and messages in real time. When an incoming call is received, the call contents are converted into text using voice recognition technology. At the same time, text messages received via messaging applications and social networking applications are also monitored. For example, if a fraudster calls Person A, the device immediately converts the call contents into text and begins analyzing it.

[0476] 4. Analysis of call content and emotions

[0477] The device sends the text of the call and messages to an analysis engine, which detects suspicious keywords and phrases. It also uses an emotion engine to analyze the user's emotions. If it determines that fraud is suspected, it sends the analysis results and the call content to the server. For example, if the device detects phrases from the call content that are highly suspicious of fraud and determines that the user is feeling anxious, it sends the content to the server.

[0478] 5. Detailed analysis and response generation by the server

[0479] The server performs a detailed analysis based on the call content received from the device and the analysis results. If it determines that fraud is suspected using generative AI, it generates an appropriate response text. If the user's emotions are negative, it adjusts the content of the response text to match their emotions. A voice generation model is used to create a voice response based on the generated response text, and the voice data is sent to the device. For example, if the server analyzes the content of the call and determines that it contains fraud keywords such as "bank transfer," and the user is feeling anxious, it will generate a response including expressions such as "stay calm" and send it to the device.

[0480] 6. Speech generation and response

[0481] The device uses the received response text to generate a voice response to the call or message in question, instantly responding to the call or message. This response allows for effective countermeasures against scammers. If the user is extremely stressed, the call will be automatically terminated. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the device will convey this to the scammer. At the same time, if the user is under high stress, the call will be disconnected.

[0482] This series of processes allows users to quickly and effectively deal with even complex and sophisticated fraud methods. Furthermore, the system can monitor not only phone call content but also text messages on messaging and social networking applications in real time, providing similar countermeasures. By combining it with an emotion engine, the aim is to reduce the psychological burden on users and achieve safer and more secure communication.

[0483] The processing flow will be explained below.

[0484] Step 1: The user installs and launches the app on their smartphone. The user enters their account information and provides data for identity verification. At this time, the user also records their voice to register. For example, an elderly person named Mr. A installs the app, enters account information such as his name and phone number, and records his voice.

[0485] Step 2: The server receives the account information and voice data sent by the user and stores them in a database. Next, it analyzes the user's voice to generate an individual voice model. The generated voice model and initial setting data are sent to the device. For example, the server analyzes person A's voice to generate a voice model and sends that data to the smartphone.

[0486] Step 3: The device stores the initial setup data received from the server and prepares the app for operation. The voice model and various settings are activated on the device, and the app goes into standby mode. For example, the device receives the initial setup data and enables the analysis engine and speech generation engine.

[0487] Step 4: When the user receives a call, the device detects the incoming call and converts the call content into text using voice recognition technology. At the same time, text messages received via messaging and social networking applications are also monitored. For example, if a fraudster calls Person A, the device will immediately convert the call content into text and begin analyzing it.

[0488] Step 5: The device sends the converted call content and messages to an analysis engine, which detects suspicious keywords and phrases. It also uses an emotion engine to analyze the user's emotions. If it determines that fraud is suspected, it sends the analysis results and the call content to the server. For example, if the device detects phrases from the call content that are highly suspicious of fraud and determines that the user is feeling anxious, it sends the content to the server.

[0489] Step 6: The server performs a detailed analysis based on the call content received from the device and the analysis results. If it determines using generative AI that fraud is suspected, it generates an appropriate response text. If the user's emotions are negative, the content of the response text is adjusted to match the emotion. For example, if the server analyzes the call content and determines that it contains fraud keywords such as "transfer," and the user is feeling anxious, it will generate a response including expressions such as "remain calm" and send it to the device.

[0490] Step 7: The server creates a voice response using the speech generation model based on the generated response text and sends the voice data to the device. For example, the server generates a response such as "This call is suspected to be fraudulent. We will report this to the police" and sends it to the device as voice data.

[0491] Step 8: The device sends the received voice response to the other party and responds to the fraudster with a generated voice. This response can prevent fraud before it occurs. If the user is extremely stressed, the call can be automatically terminated. For example, if the device tells the fraudster in a generated voice, "This call is suspected to be fraud. We will report this to the police," and at the same time detects that the user is stressed, the call will be disconnected.

[0492] Step 9: After the call, the user can check the call history and analysis results within the app. This allows the user to confirm whether the call they received was safe. For example, if Person A checks the app after the call ends and learns from the analysis results that there was a suspicion of fraud.

[0493] Example 2

[0494] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0495] Fraud targeting the elderly is becoming more sophisticated every year, and is often difficult to prevent using conventional methods. Furthermore, real-time responses are required when a user is scammed, but existing technologies make it difficult to provide immediate responses. Furthermore, elderly users, who are more likely to fall victim to fraud, often experience anxiety and stress, and methods to alleviate this stress are also needed. Existing technologies lack systems that provide automated responses that take the user's emotional state into account, so there is an urgent need to develop a system that can reduce the psychological burden and prevent fraud quickly and effectively.

[0496] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0497] In this invention, the server includes means for acquiring user account information, means for analyzing the user's voice to generate a voice model, means for converting call content into text in real time, means for analyzing the converted text to determine whether a call is suspected of fraud, means for transmitting suspicious call content to the server, means for generating appropriate response text and creating a voice response using a generative AI model if a fraud is determined to be suspected, emotion recognition means for analyzing the user's emotions in real time, and means for automatically terminating the call based on the user's negative emotions. This makes it possible to detect calls or messages that are suspected of being fraudulent in real time and respond immediately. Furthermore, automatically terminating calls taking into account the user's emotional state significantly reduces psychological burden.

[0498] The "means for obtaining user account information" is a system component for obtaining personal information such as a user's name, phone number, email address, etc. through an application installed on a communication device.

[0499] The "server means for analyzing the user's voice and generating a voice model" is a system component on the server side that analyzes the voice data provided by the user and generates a voice model based on the voice characteristics.

[0500] "Means for converting call content into text in real time" refers to technology for converting voice data of incoming and outgoing calls into text data in real time.

[0501] The "means for analyzing the converted text to determine suspicion of fraud" is an analytical engine for analyzing the text content of the call and determining whether there is suspicion of fraud.

[0502] The "means for sending suspicious call content to a server" is a system component for sending the call content to a server if it is determined to be suspicious of fraud.

[0503] A "server means using a generative AI model" is a server-side system component that utilizes generative AI technology to generate appropriate response text and create a voice response based on that text.

[0504] "Emotion recognition means for analyzing user emotions in real time" is a technology for identifying and analyzing emotions in real time from a user's voice or text message.

[0505] The "means for automatically terminating a call based on a user's negative emotions" is a system component that automatically terminates a call when the user has negative emotions such as stress or anxiety.

[0506] This invention adds emotion recognition functionality to a system that detects and automatically responds to fraud targeting the elderly in real time. The invention includes a server means that acquires user account information through an application installed on a communication device, analyzes the user's voice, and generates a voice model.

[0507] Hardware and software used

[0508] Speech recognition technology: Uses the Google Cloud Speech-to-Text API to convert calls into text in real time.

[0509] Emotion Engine: Analyzes user emotions in real time using the Microsoft Azure Emotion API.

[0510] Database: Amazon RDS is used to store and manage user account information and voice models.

[0511] Generative AI: Uses OpenAI's GPT-3 to generate appropriate response text when suspected fraud is detected.

[0512] Call Management System: Uses Asterisk to capture real-time call data and perform text transcription.

[0513] Process Overview

[0514] First, the user installs and launches the app on their smartphone. They enter their user account information and provide data for identity verification. At this time, the user also records their voice to register. For example, elderly person A installs the app, enters account information such as their name and phone number, and records their voice.

[0515] The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice to generate an individual voice model. The generated voice model and initial setting data are sent to the device. For example, the server analyzes person A's voice to generate a voice model and sends that data to the smartphone.

[0516] The device constantly monitors calls and messages in real time. When an incoming call is received, the call contents are converted into text using voice recognition technology. At the same time, text messages received via messaging applications and social networking applications are also monitored. For example, if a fraudster calls Person A, the device immediately converts the call contents into text and begins analyzing it.

[0517] The device sends the text of the call and messages to an analysis engine, which detects suspicious keywords and phrases. It also uses an emotion engine to analyze the user's emotions. If it determines that fraud is suspected, it sends the analysis results and the call content to the server. For example, if the device detects phrases from the call content that are highly suspicious of fraud and determines that the user is feeling anxious, it sends the content to the server.

[0518] The server performs a detailed analysis based on the call content received from the device and the analysis results. If it determines that fraud is suspected using generative AI, it generates an appropriate response text. If the user's emotions are negative, it adjusts the content of the response text to match their emotions. A voice generation model is used to create a voice response based on the generated response text, and the voice data is sent to the device. For example, if the server analyzes the content of the call and determines that it contains fraud keywords such as "bank transfer" and the user is feeling anxious, it will generate a response including expressions such as "stay calm" and send it to the device.

[0519] The device uses the received response text to generate a voice response to the call or message in question, instantly responding to the call or message. This response allows for effective countermeasures against scammers. If the user is extremely stressed, the call will be automatically terminated. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the device will convey this to the scammer. At the same time, if the user is under high stress, the call will be disconnected.

[0520] Specific prompt sentence examples and output examples

[0521] Prompt statement:

[0522] User says: "This is from your bank. We're having trouble with your account..."

[0523] Analysis results:

[0524] Fraud suspicion level: High

[0525] Sentiment analysis: Determines that the user's emotion is "anxiety"

[0526] Generates the following response:

[0527] This call is suspected to be fraudulent and I will be reporting the details to the police.

[0528] Example output (response after speech generation):

[0529] This call is suspected to be a scam. I will report the details to the police. Please remain calm.

[0530] This invention allows users to prevent fraud and alleviate psychological stress through emotion recognition. It provides a system that monitors calls and analyzes emotions in real time, allowing for quick and effective response.

[0531] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0532] Step 1:

[0533] A user installs and launches an app on their smartphone, which includes downloading and installing it from the app store and then tapping the app's icon to launch it.

[0534] Input: User terminal operation.

[0535] Output: The app is launched.

[0536] Step 2:

[0537] Users enter their user account information within the app and provide data to verify their identity, such as their name, phone number, and email address, and record and register their own voice to create a voice model.

[0538] Input: Name, phone number, email address, recorded voice data.

[0539] Output: Completed user account information and audio data.

[0540] Step 3:

[0541] The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice using speech recognition technology (Google Cloud Speech-to-Text API) to generate a voice model. The generated voice model and initial setting data are sent to the device.

[0542] Input: User account information, recorded voice data.

[0543] Output: User information stored in the database. Generated speech model and initial configuration data.

[0544] Step 4:

[0545] The device goes into constant monitoring mode and monitors calls and messages in real time. When an incoming call is received, the conversation is captured and converted into text using the Google Cloud Speech-to-Text API. Similarly, text messages received via messaging and social networking applications are also monitored.

[0546] Input: Calls and text messages.

[0547] Output: Transcript of calls and monitoring records.

[0548] Step 5:

[0549] The device sends the transcribed conversations and messages to an analytics engine that detects suspicious keywords and phrases and analyzes the user's emotions using the Microsoft Azure Emotion API.

[0550] Input: Texted calls and messages.

[0551] Output: Detected suspicious keywords and phrases, analyzed sentiment data.

[0552] Step 6:

[0553] The device sends the analysis results to the server. The server performs a detailed analysis based on the received analysis results and the content of the call. If it determines that fraud is suspected using generative AI (GPT-3), it generates an appropriate response text. If the user's sentiment is judged to be negative, the content of the response text is adjusted.

[0554] Input: Analysis results, emotion data.

[0555] Output: Detailed analysis results, generated response text.

[0556] Step 7:

[0557] The server creates a voice response using a speech generation model based on the generated response text, and sends the generated voice data to the terminal.

[0558] Input: The response text.

[0559] Output: The generated audio data.

[0560] Step 8:

[0561] The device responds to calls and messages in real time using the voice data it receives, and automatically ends calls if the user is feeling extremely stressed.

[0562] Input: The generated audio data.

[0563] Output: Answered voice message, call ended.

[0564] This process effectively protects users from fraud targeting the elderly, and real-time monitoring and sentiment analysis enable swift and appropriate responses.

[0565] (Application example 2)

[0566] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0567] Fraud targeting the elderly is increasing year by year, and the methods are becoming more sophisticated and ingenious. In particular, fraud perpetrated via telephone or messaging services has the potential to cause significant damage if not dealt with promptly. Furthermore, victims often experience significant emotional stress and anxiety when they encounter fraud. However, current fraud prevention systems lack the ability to address these emotional needs. Therefore, the present invention aims to provide a more effective and secure fraud prevention system for the elderly by combining fraud detection and emotion recognition.

[0568] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server is installed in the communication device and includes means for acquiring user account information, means for analyzing the user's voice to generate a voice model, and means for converting the content of the call into text in real time. This includes means for generating appropriate response text and voice response while taking the user's emotional state into consideration when fraud is suspected, and means for adjusting the content of the response text if the user's emotions are negative and automatically terminating the call if the user's stress level is high. This not only detects fraudulent acts but also enables fraud prevention measures that reduce the mental burden on elderly people and allow them to use the service with peace of mind.

[0569] "Communication device" refers to a device that transmits and receives information using telecommunications technology, including smartphones and tablets.

[0570] "User account information" refers to data containing a user's personal information and authentication information, such as name, phone number, and password.

[0571] A "voice model" is a computer model generated by analyzing the vocal characteristics of a particular user and used to reproduce the user's voice patterns.

[0572] "Server means" refers to a collection of server devices and server programs for providing specific functions and services.

[0573] "Means for converting to text" refers to the technology or device that converts voice data into text data. A voice recognition engine is an example of this.

[0574] "Means for analyzing text to determine suspected fraud" refers to technologies or programs that analyze converted character data and assess the likelihood of fraud.

[0575] "Emotion engine" refers to algorithms or software that identify a user's emotional state from speech or text.

[0576] "Negative emotions" refers to negative emotional states such as anxiety, stress, anger, and sadness.

[0577] "Means for generating appropriate response text" refers to technology or a program that creates an appropriate message for the user or the other party based on the analysis results.

[0578] "Voice response" refers to generating voice data that reads out text data.

[0579] "Means for automatically terminating a call" refers to a mechanism or technology that forcibly terminates a call when certain conditions are met.

[0580] This invention provides a system for detecting and automatically combating fraud targeting seniors in real time through an application installed on a communication device. The invention includes the following main components:

[0581] User interface operations

[0582] First, the user installs and launches the application on their smartphone. They must enter their account information and provide data for identity verification. At this point, the user registers by recording their voice. For example, an elderly person named Mr. A installs the app, enters account information such as his name and phone number, and records his voice.

[0583] Server sends initial configuration data

[0584] The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice to generate an individual voice model. The generated voice model and initial setting data are sent to the device. For example, the server analyzes person A's voice to generate a voice model and sends that data to the smartphone.

[0585] Call and message monitoring

[0586] The device constantly monitors calls and messages in real time. When an incoming call is received, the contents of the call are converted into text using voice recognition technology. At the same time, text messages received via messaging applications and social networking applications are also monitored. For example, if Person A receives a call from a fraudster, the device immediately converts the contents of the call into text and begins analysis.

[0587] Call content and emotion analysis

[0588] After the call contents are converted into text, they are sent to an analysis engine, which detects suspicious keywords and phrases. The emotion engine also analyzes the user's emotions. If fraud is suspected, the analysis results and the contents of the call are sent to the server. For example, if the device detects phrases from the call contents that are highly suspicious of fraud and determines that the user is feeling anxious, the content is sent to the server.

[0589] Detailed analysis and response generation by the server

[0590] The server performs a detailed analysis based on the call content received from the device and the analysis results. If it determines that fraud is suspected using a generative AI model, it generates an appropriate response text. If the user's emotions are negative, it adjusts the content of the response text to match the emotion. Based on the generated response text, it creates a voice response using a voice generation model, and sends the voice data to the device. For example, if the server analyzes the content of the call and determines that it contains fraud keywords such as "bank transfer," and the user is feeling anxious, it generates a response including the expression "stay calm" and sends it to the device.

[0591] Speech generation and response

[0592] The device uses the received response text to generate a voice response to the call or message in question, instantly responding to the call or message. This response allows for effective countermeasures against scammers. If the user is extremely stressed, the call will be automatically terminated. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the device will convey this to the scammer. At the same time, if the user is under high stress, the call will be disconnected.

[0593] Examples and prompts

[0594] For example, consider the case where elderly person A receives a phone call that is suspected to be a scam. Speech recognition recognizes A's statement that "I need to transfer money," and emotion analysis determines that the stress score is high (e.g., 0.9). The fraud detection model detects the suspicion of fraud from the keyword "transfer," and the response generation model creates an appropriate response such as, "This call may be a scam. Please remain calm and do not respond to the caller's requests. The call will end."

[0595] Example prompt for a generative AI model:

[0596] We provide the call content so you can determine the emotional state: "I need to transfer money."

[0597] This invention will enable elderly people to prevent themselves from falling victim to fraud, reduce their mental burden, and live their daily lives with peace of mind.

[0598] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0599] Step 1:

[0600] User interface operations:

[0601] A user installs and launches an application on a communication device. The user enters account information (such as name and phone number) and records their voice to register. At this time, the communication device collects the entered account information and voice data and sends them to a server. The input data is the user's personal information and voice data, and the output is initial setting data sent to the server.

[0602] Step 2:

[0603] Server sends initialization data:

[0604] The server receives the account information and voice data sent by the user and stores them in a database. The server then analyzes the received voice data to generate a voice model. This voice model reproduces the user's voice characteristics. The generated voice model and initial setting data are sent to the communication device. The data processing required in this process is voice analysis, and the output is a voice model and initial setting data.

[0605] Step 3:

[0606] Call and message monitoring:

[0607] The communication device monitors all calls and messaging application interactions in real time. When an incoming call is received, the call content is converted into text using voice recognition technology. It also monitors text messages from messaging applications and social networking applications. In this monitoring process, the input is voice data and text data, and the output is the converted call content.

[0608] Step 4:

[0609] Call content and sentiment analysis:

[0610] The communication device sends the text of the call and messages to an analysis engine to detect keywords and phrases that may indicate fraud. It also uses an emotion engine to analyze the user's emotional state. If it determines that fraud is suspected, it sends that information to a server. In this step, the input is the text of the call, and the output is the analysis result and the emotional state.

[0611] Step 5:

[0612] Detailed analysis and response generation by the server:

[0613] The server performs a detailed analysis based on the call content and analysis results sent from the communication device. If suspicion of fraud is confirmed, the server uses a generative AI model to generate an appropriate response text. It also adjusts the response content based on the user's emotional state. It creates a voice response using a speech generation model based on the generated response text, and sends the voice data to the communication device. The inputs to this step are the analysis results and the emotional state, and the output is voice response data.

[0614] Step 6:

[0615] Speech generation and response:

[0616] The communication device generates a voice based on the response text received from the server and immediately responds to the caller or message recipient. It also automatically ends the call if the user is extremely stressed. In this process, the input is the voice response data and the output is the end of the call or message.

[0617] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0618] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0619] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0620] [Third embodiment]

[0621] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0622] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0623] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0624] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0625] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0626] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0627] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0628] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0629] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0630] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0631] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0632] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0633] This invention is a system that detects and automatically responds to fraud targeting elderly people in real time through an application installed on a communication device. The system includes server means for acquiring user account information and analyzing the user's voice to generate a voice model. It also includes means for converting call content into text in real time and analyzing the text to determine whether it is suspected of fraud. Suspicious call content is sent to the server, where it is analyzed in detail, and appropriate response text and voice responses are generated and sent to the terminal. This allows users to prevent fraud before it occurs. The system also has the ability to monitor text messages in messaging applications and social networking applications in real time and analyze suspected fraud.

[0634] Program processing overview and explanation

[0635] 1. User interface operation

[0636] First, the user installs and launches the app on their smartphone. They enter their user account information and provide data for identity verification. At this time, the user also records their voice to register. For example, elderly person A installs the app, enters account information such as their name and phone number, and records their voice.

[0637] 2. Server sends initial configuration data

[0638] The server receives the account information and voice data sent by the user, analyzes the user's voice, and generates a voice model. It then sends the initial setup data to the device. For example, the server analyzes person A's voice, generates a voice model, and sends that data to the smartphone.

[0639] 3. Monitor calls and messages

[0640] The device constantly monitors calls and messages in real time. When an incoming call is received, the content of the call is converted into text using voice recognition technology and sent to the analysis engine. For example, if a fraudster calls Person A, the device immediately converts the content of the call into text and begins analysis.

[0641] 4. Sending suspicious verdicts

[0642] The device analyzes the text of the call and detects any elements that may be fraudulent. If fraud is suspected, the device sends the call content to the server. For example, the device may detect phrases that are highly likely to be fraudulent from the call content and send the content to the server.

[0643] 5. Detailed analysis by the server

[0644] The server further analyzes the content of the call sent from the terminal, and if it determines that there is a high suspicion of fraud, it generates an appropriate response text. It uses that text to create a voice response and sends it to the terminal. For example, if the server analyzes the content of the call and determines that it contains a fraud keyword such as "bank transfer," it generates an appropriate response and sends it to the terminal.

[0645] 6. Speech generation and response

[0646] The device uses the received response text to generate a voice response to the call or message, allowing for effective countermeasures against fraudsters. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the device will convey this to the fraudster.

[0647] This series of processes allows users to quickly and effectively deal with even complex and sophisticated fraud methods.In addition, the system can monitor not only the content of phone calls but also text messages on messaging and social networking applications in real time, providing similar countermeasures.

[0648] The processing flow will be explained below.

[0649] Step 1: The user installs the app on their smartphone and launches it. After launching, they enter their user account information, provide identity verification data, and record their voice to register.

[0650] Step 2: The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice to generate an individual voice model. It then sends the generated voice model and initial setting data to the device.

[0651] Step 3: The device stores the initial configuration data received from the server and prepares the app for operation. The voice model and various settings are activated on the device, and the app goes into standby mode.

[0652] Step 4: When the user receives a call, the device detects the incoming call and converts the call content into text in real time using voice recognition technology. At the same time, text messages received via messaging and social networking applications are also monitored.

[0653] Step 5: The device sends the text of the call and messages to an analysis engine, which detects suspicious keywords and phrases. If fraud is suspected, the analysis results and the call content are sent to a server.

[0654] Step 6: The server performs a detailed analysis based on the call content and analysis results received from the device. If it determines that fraud is suspected, it uses the generation AI to generate an appropriate response text.

[0655] Step 7: The server creates a voice response using the voice generation model based on the generated response text, and sends the voice data to the terminal.

[0656] Step 8: The device sends the received voice response to the other party and responds to the fraudster with the generated voice, thereby preventing fraud before it occurs.

[0657] Step 9: After the call ends, the user can check the call history and analysis results within the app, allowing the user to verify whether the call they received was secure.

[0658] Example 1

[0659] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0660] Fraud targeting the elderly is becoming more sophisticated every year, and preventing such damage requires real-time monitoring of call content and rapid response. However, existing systems lack the ability to perform real-time analysis during calls or to promptly notify users, making it difficult to completely prevent fraud. The purpose of this invention is to solve these problems and provide a system that allows elderly people to make calls and send messages with peace of mind.

[0661] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0662] In this invention, the server includes a means for analyzing the user's voice to generate a voice model, a means for analyzing the converted text to determine whether fraud is suspected, and a means for generating an appropriate response text and creating a voice response if fraud is suspected. This allows the server to monitor call content in real time, and if suspicious call content is detected, immediately notify the user of the content and take appropriate measures. It also has a function for notifying the user of suspected fraud when the call ends, allowing elderly people to be quickly protected from the risk of fraud. It is also possible to monitor text messages from messaging applications and social networking service applications in real time and issue a warning if fraud is suspected.

[0663] "Communication device" is a general term for electronic devices that can make calls and send messages, and specifically includes smartphones and tablets.

[0664] "Means for obtaining user account information" refers to the function for collecting personal information entered by the user through the application.

[0665] The "server means for generating a voice model" is a server that has the function of analyzing voice characteristics based on the user's voice data and creating an individual voice profile.

[0666] "Means for converting to text" refers to speech recognition technology that converts voice data into text data in real time.

[0667] "Means for determining suspected fraud" refers to algorithms or engines that analyze text data and determine whether it is likely to be fraudulent.

[0668] "Means for transmitting to a server" refers to a communication function for transmitting data from a terminal to a server.

[0669] The "server means for generating response text and creating a voice response" is a server that has the functionality to generate response text and convert it into voice as an appropriate countermeasure in cases where there is a possibility of fraud.

[0670] "Means for transmitting to the other party" refers to a communication function for transmitting the generated voice response to the other party.

[0671] "Means for notifying" refers to a feature that notifies the user of suspected fraud when the call ends.

[0672] "Warning measures" refers to a function that warns users when a call or message suspected of being fraudulent is detected.

[0673] "Messaging applications and social networking service applications" is a general term for software for sending and receiving text messages in real time, and examples include chat apps and SNS.

[0674] "Converted text" refers to content that has been converted from speech to text using speech recognition technology.

[0675] The present invention is a system for detecting fraud targeting elderly people in real time and automatically dealing with it via an application installed on a communication device. Detailed embodiments of this system are described below.

[0676] First, the user installs a dedicated application on a communication device such as a smartphone. After installation, the user launches the application and enters user account information (such as name, phone number, and email address). Voice data is also recorded and sent to the server along with the account information. This voice data is used to verify the user's identity and to generate a voice model, as described below.

[0677] The server receives the account information and voice data sent by the user. This receiving process is performed using cloud services such as Microsoft Azure and Amazon Web Services (AWS) to ensure high security. The server then analyzes the voice data using voice analysis technologies such as Google Cloud Speech-to-Text and Amazon Polly to generate a voice model specific to the user. This generated voice model and the necessary initial configuration data are then sent to the user's device, ready for real-time voice recognition and monitoring.

[0678] The device then constantly monitors calls and messaging in real time. When a user initiates a call, the device converts the call content into text in real time using voice recognition technology (such as the Google Cloud Speech-to-Text API). This allows the call content to be captured in text format and sent to the analysis engine.

[0679] An analysis engine built into the device (for example, natural language processing technology using Python's NLTK library) analyzes the text of the call and determines whether there is any suspicion of fraud. If it is determined to be suspicious, the call is immediately sent to the server.

[0680] The server then performs further analysis to determine whether the transaction is fraudulent, using a generative AI model (e.g., OpenAI's GPT-4). If fraud is determined to be highly suspected, the server generates an appropriate response text and converts it into a voice response using speech generation technology such as the Google Text-to-Speech API.

[0681] The generated voice response is sent to the device, which then immediately plays it back to the scammer. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the scammer can be informed immediately.

[0682] Furthermore, when the call ends, the device will notify the user that a fraud attempt was suspected. The notification will include a message such as, "There was a suspected fraudulent call. Would you like to check the details?" This notification allows elderly users to quickly understand the risk of fraud and take necessary measures.

[0683] The system not only monitors the content of phone calls but also text messages in messaging apps and social networking applications (SNS) in real time, and performs similar analysis and countermeasures. Similarly, if a suspicious message is detected, it will issue a warning to the user and prompt them to take necessary measures.

[0684] Examples of concrete examples and prompts

[0685] For example, if user A installs the app, registers his account information and voice recording, and then receives a call from a fraudster saying, "Please transfer XXX million yen by today," the contents of the call are converted into text and analyzed in real time. If any suspicious content is detected, the server immediately generates an appropriate countermeasure and sends it to A's device. The device then automatically plays a voice message saying, "This call is suspected to be fraudulent," warning the fraudster.

[0686] Example prompt sentence:

[0687] Please provide specific examples of voice responses that will be triggered when a user receives a call and it is determined to be fraudulent.

[0688] In this way, the system reduces the risk of seniors being scammed in real time, helping them to communicate with peace of mind in their daily lives.

[0689] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0690] Step 1:

[0691] The user installs and launches a dedicated application on their smartphone. The user enters account information such as their name, phone number, and email address, and records voice data. The recorded voice data and account information are sent to the server by the application. The input here is the user's personal information and voice data, and the output is the transmission of information to the server.

[0692] Step 2:

[0693] The server receives the account information and voice data sent by the user. The server uses voice analysis technologies such as Google Cloud Speech-to-Text and Amazon Polly to analyze the voice data and generate a voice model. This voice model is necessary for the subsequent speech recognition process. The input is the user's voice data, and the output is the generated voice model.

[0694] Step 3:

[0695] The server sends the generated voice model and initial setting data to the user's device. The device receives this and prepares to start monitoring calls and messaging. The input is the voice model and initial setting data, and the output is sending the setting data to the device.

[0696] Step 4:

[0697] When a user initiates a call, the device converts the call content into text in real time using speech recognition technology. As a specific example, we use the Google Cloud Speech-to-Text API. The input is the call audio, and the output is the text of the call.

[0698] Step 5:

[0699] The device sends the text of the call to an analysis engine (for example, Python's NLTK library) to determine whether or not there is suspicion of fraud. The input is the text of the call, and the output is the judgment result on whether or not there is suspicion of fraud.

[0700] Step 6:

[0701] If the terminal determines that fraud is suspected, it sends the contents of the call to the server. The input is text data that is suspected of fraud, and the output is data transmission to the server.

[0702] Step 7:

[0703] If the server performs a detailed analysis and determines that there is a high probability of fraud, it generates a response text and creates a voice response. An appropriate response is generated using a generative AI model (e.g., OpenAI's GPT-4) and converted into voice using the Google Text-to-Speech API. The input is the text data suspected of fraud, and the output is the generated voice response.

[0704] Step 8:

[0705] The server sends the generated voice response to the terminal, where the input is the generated voice response and the output is the voice response sent to the terminal.

[0706] Step 9:

[0707] The terminal plays the received voice response and immediately responds to the fraudster. It also notifies the user of the suspected fraud when the call ends. The input is the received voice response data and the call end event, and the output is the voice playback and a notification to the user.

[0708] This series of steps enables the system to detect fraud targeting the elderly in real time and take prompt and appropriate action. The specific actions taken at each processing step and the details of their inputs and outputs effectively protect users from the risk of fraud and enable them to communicate with others in a safe and secure manner.

[0709] (Application example 1)

[0710] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0711] The number of sophisticated frauds targeting the elderly, such as bank transfer scams, is on the rise, making it difficult to quickly and effectively detect and prevent them using conventional methods. Furthermore, frauds are often carried out via voice calls, text messages, and social media, and the lack of a system for monitoring and dealing with these in real time is a problem. Furthermore, when fraud is suspected, it is necessary to quickly issue a warning to family members and related parties.

[0712] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0713] In this invention, the server includes means for acquiring user account information, means for analyzing the user's voice to generate a voice model, means for converting the call content to text in real time, means for analyzing the converted text to determine whether fraud is suspected, means for transmitting the suspicious call content to the server, means for generating an appropriate response text and creating a voice response if fraud is suspected, means for transmitting the generated voice response to the caller, means for issuing a warning notification to family members or related parties if fraud is suspected, and means for monitoring text messages in messaging applications and SNS applications in real time and analyzing whether fraud is suspected. This makes it possible to prevent elderly people from becoming victims of fraud and to quickly and effectively detect and deal with fraudulent activities.

[0714] A "communication device" is an electronic device such as a mobile phone or smartphone that is capable of digital communication.

[0715] "User account information" is data including the user's personal information and authentication information.

[0716] A "voice model" is a database or algorithm that features a user's voice.

[0717] "Server means" refers to a computer and its software set up to process specific data.

[0718] "Means for converting call content into text in real time" refers to speech recognition technology for converting voice into text data.

[0719] The "means for analyzing the converted text to determine suspected fraud" is an algorithm that analyzes the character data to detect fraudulent patterns.

[0720] The "means for transmitting suspicious call content to a server" is a communication technology for uploading suspicious data to a server based on the analysis results.

[0721] The "server means for generating appropriate response text and creating a voice response" is a technology for generating text and converting it into voice so that the server can take appropriate measures against suspicious call content.

[0722] The "means for transmitting the generated voice response to the other party" is a communication technique for transmitting the generated voice data to the other party during the call.

[0723] "Means for sending warning notices to family or related parties" refers to communication technologies for sending warnings to family or related parties in the event of suspected fraud.

[0724] The "means for monitoring text messages on messaging applications and social networking applications in real time" refers to technology for monitoring instant messages and text messages on social networks in real time.

[0725] "Suspected fraud analysis methods" are algorithms that analyze text messages and phone calls to detect fraudulent patterns.

[0726] In this invention, the system for detecting and automatically dealing with fraudulent acts targeting the elderly is implemented using the following hardware and software.

[0727] Hardware Configuration

[0728] Communication devices: Using digital communication devices such as smartphones, smart glasses, head-mounted displays, and domestic robots.

[0729] Server: The server consists of a computer and its software for data processing and analysis.

[0730] Software Configuration

[0731] Speech recognition software: Uses the speech_recognition library to convert speech to text.

[0732] Text analysis software: Uses the TextBlob library to analyze text data to detect suspected fraud.

[0733] Communication Method: Alert notifications are sent via email using the smtplib library.

[0734] Processing flow

[0735] 1. Obtain user account information and generate a speech model:

[0736] A user installs the application on their communication device, enters their account information, records and saves their voice, and the server analyzes the voice and generates a voice model.

[0737] 2. Real-time call monitoring and analysis:

[0738] The communications device constantly monitors calls in real time and converts them into text using speech recognition software, which is then sent to a server where text analysis software determines whether the call is suspected of fraud.

[0739] 3. Dealing with suspicious calls:

[0740] If fraud is suspected based on the analysis results, the system generates an appropriate response text and creates a voice response, which is then sent to the caller, while simultaneously activating communication methods to send warning notifications to households and relevant parties.

[0741] 4. Messaging and Social Media Application Monitoring:

[0742] The communication device monitors text messages in messaging and social networking applications in real time, analyzes the text to detect potential fraud, and similarly sends the data to a server that generates an appropriate response.

[0743] Specific examples

[0744] For example, suppose an elderly person named Mr. A has a smartphone and a fraudster calls him. In this case, the smartphone monitors the contents of the call in real time, converts the voice to text, and if the text contains a fraud keyword such as "transfer." The system immediately sends the content to the server, which analyzes it in detail and automatically generates a response such as "This call is suspected to be fraudulent," warning Mr. A. A warning notification email is also sent to Mr. A's family and associates.

[0745] Prompt Sentence Examples

[0746] For generative AI models, an example prompt might be something like:

[0747] Please create a Python program that recognizes the voice recordings made by users and determines whether the recordings contain keywords that may be fraudulent, such as "transfer" or "remittance." If a keyword is detected, please include a function that notifies the relevant parties by email.

[0748] In this way, the system of the present invention can protect seniors from fraud and take action quickly and effectively.

[0749] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0750] Step 1:

[0751] A user installs an application on a communication device, enters account information, and records voice. This input includes name, phone number, and voice data. This voice data is used to generate a voice model that reflects the user's characteristics. The server analyzes the voice data, generates a personalized voice model, and sends the initialization data to the device. The output is the user's voice model and the initialization data.

[0752] Step 2:

[0753] The communication device constantly monitors the contents of calls in real time. When an incoming call is received, the device collects the contents of the call and converts them into text using voice recognition technology. The input is the voice data collected in real time, and the output is the converted text data.

[0754] Step 3:

[0755] The terminal sends the converted text data to an analysis engine to determine whether it is suspected of fraud. This analysis involves checking whether it contains specific keywords or phrases (e.g., "transfer" or "remittance"). The input is the text data, and the output is a determination of whether it is suspected of fraud.

[0756] Step 4:

[0757] If fraud is suspected, the device sends the text data to a server for further analysis. The server then further analyzes the text data and generates an appropriate response text, which includes an appropriate response to the fraudster. The input is the suspicious text data, and the output is the appropriate response text.

[0758] Step 5:

[0759] The server converts the generated response text into voice data using speech conversion technology and sends this voice data to the terminal. The terminal uses this voice data to immediately respond to the other party. The input is the response text data, and the output is the generated voice data.

[0760] Step 6:

[0761] At the same time, if fraud is suspected, the device will send a warning notification to family members or related parties. The warning notification is sent by means of email or other means and contains details of the suspected fraudulent call. The input is the fraud judgment result, and the output is a warning notification email to family members or related parties.

[0762] Step 7:

[0763] The communication device monitors text messages from messaging and social networking applications in real time and analyzes the text data. If a suspected fraudulent message is detected, it is sent to a server for further analysis. The input is the text message data collected in real time, and the output is a judgment result of whether the message is suspected to be fraudulent and an appropriate response.

[0764] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0765] This invention adds emotion recognition functionality to a system that detects and automatically responds to fraud targeting the elderly in real time through an application installed on a communication device. The invention includes server means for acquiring user account information and analyzing the user's voice to generate a voice model. It also includes means for converting call content into text in real time and analyzing the text to determine whether it is suspected of fraud. Suspicious call content is sent to the server, where it is analyzed in detail, and appropriate response text and voice responses are generated and sent to the terminal. This allows users to prevent fraud before it occurs. The system also has the ability to monitor text messages in messaging applications and social networking applications in real time and analyze suspected fraud.

[0766] Furthermore, this system incorporates an emotion engine that recognizes emotions from the user's voice and text messages during a call. Based on the user's emotions recognized by the emotion engine, the server adjusts the content of the response text. If the user's emotions are negative, such as stress or anxiety, the system has a means to automatically end the call.

[0767] Program processing overview and explanation

[0768] 1. User interface operation

[0769] First, the user installs and launches the app on their smartphone. They enter their user account information and provide data for identity verification. At this time, the user also records their voice to register. For example, elderly person A installs the app, enters account information such as their name and phone number, and records their voice.

[0770] 2. Server sends initial configuration data

[0771] The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice to generate an individual voice model. The generated voice model and initial setting data are sent to the device. For example, the server analyzes person A's voice to generate a voice model and sends that data to the smartphone.

[0772] 3. Monitor calls and messages

[0773] The device constantly monitors calls and messages in real time. When an incoming call is received, the call contents are converted into text using voice recognition technology. At the same time, text messages received via messaging applications and social networking applications are also monitored. For example, if a fraudster calls Person A, the device immediately converts the call contents into text and begins analyzing it.

[0774] 4. Analysis of call content and emotions

[0775] The device sends the text of the call and messages to an analysis engine, which detects suspicious keywords and phrases. It also uses an emotion engine to analyze the user's emotions. If it determines that fraud is suspected, it sends the analysis results and the call content to the server. For example, if the device detects phrases from the call content that are highly suspicious of fraud and determines that the user is feeling anxious, it sends the content to the server.

[0776] 5. Detailed analysis and response generation by the server

[0777] The server performs a detailed analysis based on the call content received from the device and the analysis results. If it determines that fraud is suspected using generative AI, it generates an appropriate response text. If the user's emotions are negative, it adjusts the content of the response text to match their emotions. A voice generation model is used to create a voice response based on the generated response text, and the voice data is sent to the device. For example, if the server analyzes the content of the call and determines that it contains fraud keywords such as "bank transfer," and the user is feeling anxious, it will generate a response including expressions such as "stay calm" and send it to the device.

[0778] 6. Speech generation and response

[0779] The device uses the received response text to generate a voice response to the call or message in question, instantly responding to the call or message. This response allows for effective countermeasures against scammers. If the user is extremely stressed, the call will be automatically terminated. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the device will convey this to the scammer. At the same time, if the user is under high stress, the call will be disconnected.

[0780] This series of processes allows users to quickly and effectively deal with even complex and sophisticated fraud methods. Furthermore, the system can monitor not only phone call content but also text messages on messaging and social networking applications in real time, providing similar countermeasures. By combining it with an emotion engine, the aim is to reduce the psychological burden on users and achieve safer and more secure communication.

[0781] The processing flow will be explained below.

[0782] Step 1: The user installs and launches the app on their smartphone. The user enters their account information and provides data for identity verification. At this time, the user also records their voice to register. For example, an elderly person named Mr. A installs the app, enters account information such as his name and phone number, and records his voice.

[0783] Step 2: The server receives the account information and voice data sent by the user and stores them in a database. Next, it analyzes the user's voice to generate an individual voice model. The generated voice model and initial setting data are sent to the device. For example, the server analyzes person A's voice to generate a voice model and sends that data to the smartphone.

[0784] Step 3: The device stores the initial setup data received from the server and prepares the app for operation. The voice model and various settings are activated on the device, and the app goes into standby mode. For example, the device receives the initial setup data and enables the analysis engine and speech generation engine.

[0785] Step 4: When the user receives a call, the device detects the incoming call and converts the call content into text using voice recognition technology. At the same time, text messages received via messaging and social networking applications are also monitored. For example, if a fraudster calls Person A, the device will immediately convert the call content into text and begin analyzing it.

[0786] Step 5: The device sends the converted call content and messages to an analysis engine, which detects suspicious keywords and phrases. It also uses an emotion engine to analyze the user's emotions. If it determines that fraud is suspected, it sends the analysis results and the call content to the server. For example, if the device detects phrases from the call content that are highly suspicious of fraud and determines that the user is feeling anxious, it sends the content to the server.

[0787] Step 6: The server performs a detailed analysis based on the call content received from the device and the analysis results. If it determines using generative AI that fraud is suspected, it generates an appropriate response text. If the user's emotions are negative, the content of the response text is adjusted to match the emotion. For example, if the server analyzes the call content and determines that it contains fraud keywords such as "transfer," and the user is feeling anxious, it will generate a response including expressions such as "remain calm" and send it to the device.

[0788] Step 7: The server creates a voice response using the speech generation model based on the generated response text and sends the voice data to the device. For example, the server generates a response such as "This call is suspected to be fraudulent. We will report this to the police" and sends it to the device as voice data.

[0789] Step 8: The device sends the received voice response to the other party and responds to the fraudster with a generated voice. This response can prevent fraud before it occurs. If the user is extremely stressed, the call can be automatically terminated. For example, if the device tells the fraudster in a generated voice, "This call is suspected to be fraud. We will report this to the police," and at the same time detects that the user is stressed, the call will be disconnected.

[0790] Step 9: After the call, the user can check the call history and analysis results within the app. This allows the user to confirm whether the call they received was safe. For example, if Person A checks the app after the call ends and learns from the analysis results that there was a suspicion of fraud.

[0791] Example 2

[0792] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0793] Fraud targeting the elderly is becoming more sophisticated every year, and is often difficult to prevent using conventional methods. Furthermore, real-time responses are required when a user is scammed, but existing technologies make it difficult to provide immediate responses. Furthermore, elderly users, who are more likely to fall victim to fraud, often experience anxiety and stress, and methods to alleviate this stress are also needed. Existing technologies lack systems that provide automated responses that take the user's emotional state into account, so there is an urgent need to develop a system that can reduce the psychological burden and prevent fraud quickly and effectively.

[0794] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0795] In this invention, the server includes means for acquiring user account information, means for analyzing the user's voice to generate a voice model, means for converting call content into text in real time, means for analyzing the converted text to determine whether a call is suspected of fraud, means for transmitting suspicious call content to the server, means for generating appropriate response text and creating a voice response using a generative AI model if a fraud is determined to be suspected, emotion recognition means for analyzing the user's emotions in real time, and means for automatically terminating the call based on the user's negative emotions. This makes it possible to detect calls or messages that are suspected of being fraudulent in real time and respond immediately. Furthermore, automatically terminating calls taking into account the user's emotional state significantly reduces psychological burden.

[0796] The "means for obtaining user account information" is a system component for obtaining personal information such as a user's name, phone number, email address, etc. through an application installed on a communication device.

[0797] The "server means for analyzing the user's voice and generating a voice model" is a system component on the server side that analyzes the voice data provided by the user and generates a voice model based on the voice characteristics.

[0798] "Means for converting call content into text in real time" refers to technology for converting voice data of incoming and outgoing calls into text data in real time.

[0799] The "means for analyzing the converted text to determine suspicion of fraud" is an analytical engine for analyzing the text content of the call and determining whether there is suspicion of fraud.

[0800] The "means for sending suspicious call content to a server" is a system component for sending the call content to a server if it is determined to be suspicious of fraud.

[0801] A "server means using a generative AI model" is a server-side system component that utilizes generative AI technology to generate appropriate response text and create a voice response based on that text.

[0802] "Emotion recognition means for analyzing user emotions in real time" is a technology for identifying and analyzing emotions in real time from a user's voice or text message.

[0803] The "means for automatically terminating a call based on a user's negative emotions" is a system component that automatically terminates a call when the user has negative emotions such as stress or anxiety.

[0804] This invention adds emotion recognition functionality to a system that detects and automatically responds to fraud targeting the elderly in real time. The invention includes a server means that acquires user account information through an application installed on a communication device, analyzes the user's voice, and generates a voice model.

[0805] Hardware and software used

[0806] Speech recognition technology: Uses the Google Cloud Speech-to-Text API to convert calls into text in real time.

[0807] Emotion Engine: Analyzes user emotions in real time using the Microsoft Azure Emotion API.

[0808] Database: Amazon RDS is used to store and manage user account information and voice models.

[0809] Generative AI: Uses OpenAI's GPT-3 to generate appropriate response text when suspected fraud is detected.

[0810] Call Management System: Uses Asterisk to capture real-time call data and perform text transcription.

[0811] Process Overview

[0812] First, the user installs and launches the app on their smartphone. They enter their user account information and provide data for identity verification. At this time, the user also records their voice to register. For example, elderly person A installs the app, enters account information such as their name and phone number, and records their voice.

[0813] The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice to generate an individual voice model. The generated voice model and initial setting data are sent to the device. For example, the server analyzes person A's voice to generate a voice model and sends that data to the smartphone.

[0814] The device constantly monitors calls and messages in real time. When an incoming call is received, the call contents are converted into text using voice recognition technology. At the same time, text messages received via messaging applications and social networking applications are also monitored. For example, if a fraudster calls Person A, the device immediately converts the call contents into text and begins analyzing it.

[0815] The device sends the text of the call and messages to an analysis engine, which detects suspicious keywords and phrases. It also uses an emotion engine to analyze the user's emotions. If it determines that fraud is suspected, it sends the analysis results and the call content to the server. For example, if the device detects phrases from the call content that are highly suspicious of fraud and determines that the user is feeling anxious, it sends the content to the server.

[0816] The server performs a detailed analysis based on the call content received from the device and the analysis results. If it determines that fraud is suspected using generative AI, it generates an appropriate response text. If the user's emotions are negative, it adjusts the content of the response text to match their emotions. A voice generation model is used to create a voice response based on the generated response text, and the voice data is sent to the device. For example, if the server analyzes the content of the call and determines that it contains fraud keywords such as "bank transfer" and the user is feeling anxious, it will generate a response including expressions such as "stay calm" and send it to the device.

[0817] The device uses the received response text to generate a voice response to the call or message in question, instantly responding to the call or message. This response allows for effective countermeasures against scammers. If the user is extremely stressed, the call will be automatically terminated. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the device will convey this to the scammer. At the same time, if the user is under high stress, the call will be disconnected.

[0818] Specific prompt sentence examples and output examples

[0819] Prompt statement:

[0820] User says: "This is from your bank. We're having trouble with your account..."

[0821] Analysis results:

[0822] Fraud suspicion level: High

[0823] Sentiment analysis: Determines that the user's emotion is "anxiety"

[0824] Generates the following response:

[0825] This call is suspected to be fraudulent and I will be reporting the details to the police.

[0826] Example output (response after speech generation):

[0827] This call is suspected to be a scam. I will report the details to the police. Please remain calm.

[0828] This invention allows users to prevent fraud and alleviate psychological stress through emotion recognition. It provides a system that monitors calls and analyzes emotions in real time, allowing for quick and effective response.

[0829] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0830] Step 1:

[0831] A user installs and launches an app on their smartphone, which includes downloading and installing it from the app store and then tapping the app's icon to launch it.

[0832] Input: User terminal operation.

[0833] Output: The app is launched.

[0834] Step 2:

[0835] Users enter their user account information within the app and provide data to verify their identity, such as their name, phone number, and email address, and record and register their own voice to create a voice model.

[0836] Input: Name, phone number, email address, recorded voice data.

[0837] Output: Completed user account information and audio data.

[0838] Step 3:

[0839] The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice using speech recognition technology (Google Cloud Speech-to-Text API) to generate a voice model. The generated voice model and initial setting data are sent to the device.

[0840] Input: User account information, recorded voice data.

[0841] Output: User information stored in the database. Generated speech model and initial configuration data.

[0842] Step 4:

[0843] The device goes into constant monitoring mode and monitors calls and messages in real time. When an incoming call is received, the conversation is captured and converted into text using the Google Cloud Speech-to-Text API. Similarly, text messages received via messaging and social networking applications are also monitored.

[0844] Input: Calls and text messages.

[0845] Output: Transcript of calls and monitoring records.

[0846] Step 5:

[0847] The device sends the transcribed conversations and messages to an analytics engine that detects suspicious keywords and phrases and analyzes the user's emotions using the Microsoft Azure Emotion API.

[0848] Input: Texted calls and messages.

[0849] Output: Detected suspicious keywords and phrases, analyzed sentiment data.

[0850] Step 6:

[0851] The device sends the analysis results to the server. The server performs a detailed analysis based on the received analysis results and the content of the call. If it determines that fraud is suspected using generative AI (GPT-3), it generates an appropriate response text. If the user's sentiment is judged to be negative, the content of the response text is adjusted.

[0852] Input: Analysis results, emotion data.

[0853] Output: Detailed analysis results, generated response text.

[0854] Step 7:

[0855] The server creates a voice response using a speech generation model based on the generated response text, and sends the generated voice data to the terminal.

[0856] Input: The response text.

[0857] Output: The generated audio data.

[0858] Step 8:

[0859] The device responds to calls and messages in real time using the voice data it receives, and automatically ends calls if the user is feeling extremely stressed.

[0860] Input: The generated audio data.

[0861] Output: Answered voice message, call ended.

[0862] This process effectively protects users from fraud targeting the elderly, and real-time monitoring and sentiment analysis enable swift and appropriate responses.

[0863] (Application example 2)

[0864] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0865] Fraud targeting the elderly is increasing year by year, and the methods are becoming more sophisticated and ingenious. In particular, fraud perpetrated via telephone or messaging services has the potential to cause significant damage if not dealt with promptly. Furthermore, victims often experience significant emotional stress and anxiety when they encounter fraud. However, current fraud prevention systems lack the ability to address these emotional needs. Therefore, the present invention aims to provide a more effective and secure fraud prevention system for the elderly by combining fraud detection and emotion recognition.

[0866] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server is installed in the communication device and includes means for acquiring user account information, means for analyzing the user's voice to generate a voice model, and means for converting the content of the call into text in real time. This includes means for generating appropriate response text and voice response while taking the user's emotional state into consideration when fraud is suspected, and means for adjusting the content of the response text if the user's emotions are negative and automatically terminating the call if the user's stress level is high. This not only detects fraudulent acts but also enables fraud prevention measures that reduce the mental burden on elderly people and allow them to use the service with peace of mind.

[0867] "Communication device" refers to a device that transmits and receives information using telecommunications technology, including smartphones and tablets.

[0868] "User account information" refers to data containing a user's personal information and authentication information, such as name, phone number, and password.

[0869] A "voice model" is a computer model generated by analyzing the vocal characteristics of a particular user and used to reproduce the user's voice patterns.

[0870] "Server means" refers to a collection of server devices and server programs for providing specific functions and services.

[0871] "Means for converting to text" refers to the technology or device that converts voice data into text data. A voice recognition engine is an example of this.

[0872] "Means for analyzing text to determine suspected fraud" refers to technologies or programs that analyze converted character data and assess the likelihood of fraud.

[0873] "Emotion engine" refers to algorithms or software that identify a user's emotional state from speech or text.

[0874] "Negative emotions" refers to negative emotional states such as anxiety, stress, anger, and sadness.

[0875] "Means for generating appropriate response text" refers to technology or a program that creates an appropriate message for the user or the other party based on the analysis results.

[0876] "Voice response" refers to generating voice data that reads out text data.

[0877] "Means for automatically terminating a call" refers to a mechanism or technology that forcibly terminates a call when certain conditions are met.

[0878] This invention provides a system for detecting and automatically combating fraud targeting seniors in real time through an application installed on a communication device. The invention includes the following main components:

[0879] User interface operations

[0880] First, the user installs and launches the application on their smartphone. They must enter their account information and provide data for identity verification. At this point, the user registers by recording their voice. For example, an elderly person named Mr. A installs the app, enters account information such as his name and phone number, and records his voice.

[0881] Server sends initial configuration data

[0882] The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice to generate an individual voice model. The generated voice model and initial setting data are sent to the device. For example, the server analyzes person A's voice to generate a voice model and sends that data to the smartphone.

[0883] Call and message monitoring

[0884] The device constantly monitors calls and messages in real time. When an incoming call is received, the contents of the call are converted into text using voice recognition technology. At the same time, text messages received via messaging applications and social networking applications are also monitored. For example, if Person A receives a call from a fraudster, the device immediately converts the contents of the call into text and begins analysis.

[0885] Call content and emotion analysis

[0886] After the call contents are converted into text, they are sent to an analysis engine, which detects suspicious keywords and phrases. The emotion engine also analyzes the user's emotions. If fraud is suspected, the analysis results and the contents of the call are sent to the server. For example, if the device detects phrases from the call contents that are highly suspicious of fraud and determines that the user is feeling anxious, the content is sent to the server.

[0887] Detailed analysis and response generation by the server

[0888] The server performs a detailed analysis based on the call content received from the device and the analysis results. If it determines that fraud is suspected using a generative AI model, it generates an appropriate response text. If the user's emotions are negative, it adjusts the content of the response text to match the emotion. Based on the generated response text, it creates a voice response using a voice generation model, and sends the voice data to the device. For example, if the server analyzes the content of the call and determines that it contains fraud keywords such as "bank transfer," and the user is feeling anxious, it generates a response including the expression "stay calm" and sends it to the device.

[0889] Speech generation and response

[0890] The device uses the received response text to generate a voice response to the call or message in question, instantly responding to the call or message. This response allows for effective countermeasures against scammers. If the user is extremely stressed, the call will be automatically terminated. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the device will convey this to the scammer. At the same time, if the user is under high stress, the call will be disconnected.

[0891] Examples and prompts

[0892] For example, consider the case where elderly person A receives a phone call that is suspected to be a scam. Speech recognition recognizes A's statement that "I need to transfer money," and emotion analysis determines that the stress score is high (e.g., 0.9). The fraud detection model detects the suspicion of fraud from the keyword "transfer," and the response generation model creates an appropriate response such as, "This call may be a scam. Please remain calm and do not respond to the caller's requests. The call will end."

[0893] Example prompt for a generative AI model:

[0894] We provide the call content so you can determine the emotional state: "I need to transfer money."

[0895] This invention will enable elderly people to prevent themselves from falling victim to fraud, reduce their mental burden, and live their daily lives with peace of mind.

[0896] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0897] Step 1:

[0898] User interface operations:

[0899] A user installs and launches an application on a communication device. The user enters account information (such as name and phone number) and records their voice to register. At this time, the communication device collects the entered account information and voice data and sends them to a server. The input data is the user's personal information and voice data, and the output is initial setting data sent to the server.

[0900] Step 2:

[0901] Server sends initialization data:

[0902] The server receives the account information and voice data sent by the user and stores them in a database. The server then analyzes the received voice data to generate a voice model. This voice model reproduces the user's voice characteristics. The generated voice model and initial setting data are sent to the communication device. The data processing required in this process is voice analysis, and the output is a voice model and initial setting data.

[0903] Step 3:

[0904] Call and message monitoring:

[0905] The communication device monitors all calls and messaging application interactions in real time. When an incoming call is received, the call content is converted into text using voice recognition technology. It also monitors text messages from messaging applications and social networking applications. In this monitoring process, the input is voice data and text data, and the output is the converted call content.

[0906] Step 4:

[0907] Call content and sentiment analysis:

[0908] The communication device sends the text of the call and messages to an analysis engine to detect keywords and phrases that may indicate fraud. It also uses an emotion engine to analyze the user's emotional state. If it determines that fraud is suspected, it sends that information to a server. In this step, the input is the text of the call, and the output is the analysis result and the emotional state.

[0909] Step 5:

[0910] Detailed analysis and response generation by the server:

[0911] The server performs a detailed analysis based on the call content and analysis results sent from the communication device. If suspicion of fraud is confirmed, the server uses a generative AI model to generate an appropriate response text. It also adjusts the response content based on the user's emotional state. It creates a voice response using a speech generation model based on the generated response text, and sends the voice data to the communication device. The inputs to this step are the analysis results and the emotional state, and the output is voice response data.

[0912] Step 6:

[0913] Speech generation and response:

[0914] The communication device generates a voice based on the response text received from the server and immediately responds to the caller or message recipient. It also automatically ends the call if the user is extremely stressed. In this process, the input is the voice response data and the output is the end of the call or message.

[0915] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0916] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0917] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0918] [Fourth embodiment]

[0919] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0920] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0921] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0922] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0923] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0924] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0925] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0926] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0927] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0928] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0929] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0930] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0931] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0932] This invention is a system that detects and automatically responds to fraud targeting elderly people in real time through an application installed on a communication device. The system includes server means for acquiring user account information and analyzing the user's voice to generate a voice model. It also includes means for converting call content into text in real time and analyzing the text to determine whether it is suspected of fraud. Suspicious call content is sent to the server, where it is analyzed in detail, and appropriate response text and voice responses are generated and sent to the terminal. This allows users to prevent fraud before it occurs. The system also has the ability to monitor text messages in messaging applications and social networking applications in real time and analyze suspected fraud.

[0933] Program processing overview and explanation

[0934] 1. User interface operation

[0935] First, the user installs and launches the app on their smartphone. They enter their user account information and provide data for identity verification. At this time, the user also records their voice to register. For example, elderly person A installs the app, enters account information such as their name and phone number, and records their voice.

[0936] 2. Server sends initial configuration data

[0937] The server receives the account information and voice data sent by the user, analyzes the user's voice, and generates a voice model. It then sends the initial setup data to the device. For example, the server analyzes person A's voice, generates a voice model, and sends that data to the smartphone.

[0938] 3. Monitor calls and messages

[0939] The device constantly monitors calls and messages in real time. When an incoming call is received, the content of the call is converted into text using voice recognition technology and sent to the analysis engine. For example, if a fraudster calls Person A, the device immediately converts the content of the call into text and begins analysis.

[0940] 4. Sending suspicious verdicts

[0941] The device analyzes the text of the call and detects any elements that may be fraudulent. If fraud is suspected, the device sends the call content to the server. For example, the device may detect phrases that are highly likely to be fraudulent from the call content and send the content to the server.

[0942] 5. Detailed analysis by the server

[0943] The server further analyzes the content of the call sent from the terminal, and if it determines that there is a high suspicion of fraud, it generates an appropriate response text. It uses that text to create a voice response and sends it to the terminal. For example, if the server analyzes the content of the call and determines that it contains a fraud keyword such as "bank transfer," it generates an appropriate response and sends it to the terminal.

[0944] 6. Speech generation and response

[0945] The device uses the received response text to generate a voice response to the call or message, allowing for effective countermeasures against fraudsters. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the device will convey this to the fraudster.

[0946] This series of processes allows users to quickly and effectively deal with even complex and sophisticated fraud methods.In addition, the system can monitor not only the content of phone calls but also text messages on messaging and social networking applications in real time, providing similar countermeasures.

[0947] The processing flow will be explained below.

[0948] Step 1: The user installs the app on their smartphone and launches it. After launching, they enter their user account information, provide identity verification data, and record their voice to register.

[0949] Step 2: The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice to generate an individual voice model. It then sends the generated voice model and initial setting data to the device.

[0950] Step 3: The device stores the initial configuration data received from the server and prepares the app for operation. The voice model and various settings are activated on the device, and the app goes into standby mode.

[0951] Step 4: When the user receives a call, the device detects the incoming call and converts the call content into text in real time using voice recognition technology. At the same time, text messages received via messaging and social networking applications are also monitored.

[0952] Step 5: The device sends the text of the call and messages to an analysis engine, which detects suspicious keywords and phrases. If fraud is suspected, the analysis results and the call content are sent to a server.

[0953] Step 6: The server performs a detailed analysis based on the call content and analysis results received from the device. If it determines that fraud is suspected, it uses the generation AI to generate an appropriate response text.

[0954] Step 7: The server creates a voice response using the voice generation model based on the generated response text, and sends the voice data to the terminal.

[0955] Step 8: The device sends the received voice response to the other party and responds to the fraudster with the generated voice, thereby preventing fraud before it occurs.

[0956] Step 9: After the call ends, the user can check the call history and analysis results within the app, allowing the user to verify whether the call they received was secure.

[0957] Example 1

[0958] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0959] Fraud targeting the elderly is becoming more sophisticated every year, and preventing such damage requires real-time monitoring of call content and rapid response. However, existing systems lack the ability to perform real-time analysis during calls or to promptly notify users, making it difficult to completely prevent fraud. The purpose of this invention is to solve these problems and provide a system that allows elderly people to make calls and send messages with peace of mind.

[0960] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0961] In this invention, the server includes a means for analyzing the user's voice to generate a voice model, a means for analyzing the converted text to determine whether fraud is suspected, and a means for generating an appropriate response text and creating a voice response if fraud is suspected. This allows the server to monitor call content in real time, and if suspicious call content is detected, immediately notify the user of the content and take appropriate measures. It also has a function for notifying the user of suspected fraud when the call ends, allowing elderly people to be quickly protected from the risk of fraud. It is also possible to monitor text messages from messaging applications and social networking service applications in real time and issue a warning if fraud is suspected.

[0962] "Communication device" is a general term for electronic devices that can make calls and send messages, and specifically includes smartphones and tablets.

[0963] "Means for obtaining user account information" refers to the function for collecting personal information entered by the user through the application.

[0964] The "server means for generating a voice model" is a server that has the function of analyzing voice characteristics based on the user's voice data and creating an individual voice profile.

[0965] "Means for converting to text" refers to speech recognition technology that converts voice data into text data in real time.

[0966] "Means for determining suspected fraud" refers to algorithms or engines that analyze text data and determine whether it is likely to be fraudulent.

[0967] "Means for transmitting to a server" refers to a communication function for transmitting data from a terminal to a server.

[0968] The "server means for generating response text and creating a voice response" is a server that has the functionality to generate response text and convert it into voice as an appropriate countermeasure in cases where there is a possibility of fraud.

[0969] "Means for transmitting to the other party" refers to a communication function for transmitting the generated voice response to the other party.

[0970] "Means for notifying" refers to a feature that notifies the user of suspected fraud when the call ends.

[0971] "Warning measures" refers to a function that warns users when a call or message suspected of being fraudulent is detected.

[0972] "Messaging applications and social networking service applications" is a general term for software for sending and receiving text messages in real time, and examples include chat apps and SNS.

[0973] "Converted text" refers to content that has been converted from speech to text using speech recognition technology.

[0974] The present invention is a system for detecting fraud targeting elderly people in real time and automatically dealing with it via an application installed on a communication device. Detailed embodiments of this system are described below.

[0975] First, the user installs a dedicated application on a communication device such as a smartphone. After installation, the user launches the application and enters user account information (such as name, phone number, and email address). Voice data is also recorded and sent to the server along with the account information. This voice data is used to verify the user's identity and to generate a voice model, as described below.

[0976] The server receives the account information and voice data sent by the user. This receiving process is performed using cloud services such as Microsoft Azure and Amazon Web Services (AWS) to ensure high security. The server then analyzes the voice data using voice analysis technologies such as Google Cloud Speech-to-Text and Amazon Polly to generate a voice model specific to the user. This generated voice model and the necessary initial configuration data are then sent to the user's device, ready for real-time voice recognition and monitoring.

[0977] The device then constantly monitors calls and messaging in real time. When a user initiates a call, the device converts the call content into text in real time using voice recognition technology (such as the Google Cloud Speech-to-Text API). This allows the call content to be captured in text format and sent to the analysis engine.

[0978] An analysis engine built into the device (for example, natural language processing technology using Python's NLTK library) analyzes the text of the call and determines whether there is any suspicion of fraud. If it is determined to be suspicious, the call is immediately sent to the server.

[0979] The server then performs further analysis to determine whether the transaction is fraudulent, using a generative AI model (e.g., OpenAI's GPT-4). If fraud is determined to be highly suspected, the server generates an appropriate response text and converts it into a voice response using speech generation technology such as the Google Text-to-Speech API.

[0980] The generated voice response is sent to the device, which then immediately plays it back to the scammer. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the scammer can be informed immediately.

[0981] Furthermore, when the call ends, the device will notify the user that a fraud attempt was suspected. The notification will include a message such as, "There was a suspected fraudulent call. Would you like to check the details?" This notification allows elderly users to quickly understand the risk of fraud and take necessary measures.

[0982] The system not only monitors the content of phone calls but also text messages in messaging apps and social networking applications (SNS) in real time, and performs similar analysis and countermeasures. Similarly, if a suspicious message is detected, it will issue a warning to the user and prompt them to take necessary measures.

[0983] Examples of concrete examples and prompts

[0984] For example, if user A installs the app, registers his account information and voice recording, and then receives a call from a fraudster saying, "Please transfer XXX million yen by today," the contents of the call are converted into text and analyzed in real time. If any suspicious content is detected, the server immediately generates an appropriate countermeasure and sends it to A's device. The device then automatically plays a voice message saying, "This call is suspected to be fraudulent," warning the fraudster.

[0985] Example prompt sentence:

[0986] Please provide specific examples of voice responses that will be triggered when a user receives a call and it is determined to be fraudulent.

[0987] In this way, the system reduces the risk of seniors being scammed in real time, helping them to communicate with peace of mind in their daily lives.

[0988] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0989] Step 1:

[0990] The user installs and launches a dedicated application on their smartphone. The user enters account information such as their name, phone number, and email address, and records voice data. The recorded voice data and account information are sent to the server by the application. The input here is the user's personal information and voice data, and the output is the transmission of information to the server.

[0991] Step 2:

[0992] The server receives the account information and voice data sent by the user. The server uses voice analysis technologies such as Google Cloud Speech-to-Text and Amazon Polly to analyze the voice data and generate a voice model. This voice model is necessary for the subsequent speech recognition process. The input is the user's voice data, and the output is the generated voice model.

[0993] Step 3:

[0994] The server sends the generated voice model and initial setting data to the user's device. The device receives this and prepares to start monitoring calls and messaging. The input is the voice model and initial setting data, and the output is sending the setting data to the device.

[0995] Step 4:

[0996] When a user initiates a call, the device converts the call content into text in real time using speech recognition technology. As a specific example, we use the Google Cloud Speech-to-Text API. The input is the call audio, and the output is the text of the call.

[0997] Step 5:

[0998] The device sends the text of the call to an analysis engine (for example, Python's NLTK library) to determine whether or not there is suspicion of fraud. The input is the text of the call, and the output is the judgment result on whether or not there is suspicion of fraud.

[0999] Step 6:

[1000] If the terminal determines that fraud is suspected, it sends the contents of the call to the server. The input is text data that is suspected of fraud, and the output is data transmission to the server.

[1001] Step 7:

[1002] If the server performs a detailed analysis and determines that there is a high probability of fraud, it generates a response text and creates a voice response. An appropriate response is generated using a generative AI model (e.g., OpenAI's GPT-4) and converted into voice using the Google Text-to-Speech API. The input is the text data suspected of fraud, and the output is the generated voice response.

[1003] Step 8:

[1004] The server sends the generated voice response to the terminal, where the input is the generated voice response and the output is the voice response sent to the terminal.

[1005] Step 9:

[1006] The terminal plays the received voice response and immediately responds to the fraudster. It also notifies the user of the suspected fraud when the call ends. The input is the received voice response data and the call end event, and the output is the voice playback and a notification to the user.

[1007] This series of steps enables the system to detect fraud targeting the elderly in real time and take prompt and appropriate action. The specific actions taken at each processing step and the details of their inputs and outputs effectively protect users from the risk of fraud and enable them to communicate with others in a safe and secure manner.

[1008] (Application example 1)

[1009] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1010] The number of sophisticated frauds targeting the elderly, such as bank transfer scams, is on the rise, making it difficult to quickly and effectively detect and prevent them using conventional methods. Furthermore, frauds are often carried out via voice calls, text messages, and social media, and the lack of a system for monitoring and dealing with these in real time is a problem. Furthermore, when fraud is suspected, it is necessary to quickly issue a warning to family members and related parties.

[1011] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1012] In this invention, the server includes means for acquiring user account information, means for analyzing the user's voice to generate a voice model, means for converting the call content to text in real time, means for analyzing the converted text to determine whether fraud is suspected, means for transmitting the suspicious call content to the server, means for generating an appropriate response text and creating a voice response if fraud is suspected, means for transmitting the generated voice response to the caller, means for issuing a warning notification to family members or related parties if fraud is suspected, and means for monitoring text messages in messaging applications and SNS applications in real time and analyzing whether fraud is suspected. This makes it possible to prevent elderly people from becoming victims of fraud and to quickly and effectively detect and deal with fraudulent activities.

[1013] A "communication device" is an electronic device such as a mobile phone or smartphone that is capable of digital communication.

[1014] "User account information" is data including the user's personal information and authentication information.

[1015] A "voice model" is a database or algorithm that features a user's voice.

[1016] "Server means" refers to a computer and its software set up to process specific data.

[1017] "Means for converting call content into text in real time" refers to speech recognition technology for converting voice into text data.

[1018] The "means for analyzing the converted text to determine suspected fraud" is an algorithm that analyzes the character data to detect fraudulent patterns.

[1019] The "means for transmitting suspicious call content to a server" is a communication technology for uploading suspicious data to a server based on the analysis results.

[1020] The "server means for generating appropriate response text and creating a voice response" is a technology for generating text and converting it into voice so that the server can take appropriate measures against suspicious call content.

[1021] The "means for transmitting the generated voice response to the other party" is a communication technique for transmitting the generated voice data to the other party during the call.

[1022] "Means for sending warning notices to family or related parties" refers to communication technologies for sending warnings to family or related parties in the event of suspected fraud.

[1023] The "means for monitoring text messages on messaging applications and social networking applications in real time" refers to technology for monitoring instant messages and text messages on social networks in real time.

[1024] "Suspected fraud analysis methods" are algorithms that analyze text messages and phone calls to detect fraudulent patterns.

[1025] In this invention, the system for detecting and automatically dealing with fraudulent acts targeting the elderly is implemented using the following hardware and software.

[1026] Hardware Configuration

[1027] Communication devices: Using digital communication devices such as smartphones, smart glasses, head-mounted displays, and domestic robots.

[1028] Server: The server consists of a computer and its software for data processing and analysis.

[1029] Software Configuration

[1030] Speech recognition software: Uses the speech_recognition library to convert speech to text.

[1031] Text analysis software: Uses the TextBlob library to analyze text data to detect suspected fraud.

[1032] Communication Method: Alert notifications are sent via email using the smtplib library.

[1033] Processing flow

[1034] 1. Obtain user account information and generate a speech model:

[1035] A user installs the application on their communication device, enters their account information, records and saves their voice, and the server analyzes the voice and generates a voice model.

[1036] 2. Real-time call monitoring and analysis:

[1037] The communications device constantly monitors calls in real time and converts them into text using speech recognition software, which is then sent to a server where text analysis software determines whether the call is suspected of fraud.

[1038] 3. Dealing with suspicious calls:

[1039] If fraud is suspected based on the analysis results, the system generates an appropriate response text and creates a voice response, which is then sent to the caller, while simultaneously activating communication methods to send warning notifications to households and relevant parties.

[1040] 4. Messaging and Social Media Application Monitoring:

[1041] The communication device monitors text messages in messaging and social networking applications in real time, analyzes the text to detect potential fraud, and similarly sends the data to a server that generates an appropriate response.

[1042] Specific examples

[1043] For example, suppose an elderly person named Mr. A has a smartphone and a fraudster calls him. In this case, the smartphone monitors the contents of the call in real time, converts the voice to text, and if the text contains a fraud keyword such as "transfer." The system immediately sends the content to the server, which analyzes it in detail and automatically generates a response such as "This call is suspected to be fraudulent," warning Mr. A. A warning notification email is also sent to Mr. A's family and associates.

[1044] Prompt Sentence Examples

[1045] For generative AI models, an example prompt might be something like:

[1046] Please create a Python program that recognizes the voice recordings made by users and determines whether the recordings contain keywords that may be fraudulent, such as "transfer" or "remittance." If a keyword is detected, please include a function that notifies the relevant parties by email.

[1047] In this way, the system of the present invention can protect seniors from fraud and take action quickly and effectively.

[1048] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1049] Step 1:

[1050] A user installs an application on a communication device, enters account information, and records voice. This input includes name, phone number, and voice data. This voice data is used to generate a voice model that reflects the user's characteristics. The server analyzes the voice data, generates a personalized voice model, and sends the initialization data to the device. The output is the user's voice model and the initialization data.

[1051] Step 2:

[1052] The communication device constantly monitors the contents of calls in real time. When an incoming call is received, the device collects the contents of the call and converts them into text using voice recognition technology. The input is the voice data collected in real time, and the output is the converted text data.

[1053] Step 3:

[1054] The terminal sends the converted text data to an analysis engine to determine whether it is suspected of fraud. This analysis involves checking whether it contains specific keywords or phrases (e.g., "transfer" or "remittance"). The input is the text data, and the output is a determination of whether it is suspected of fraud.

[1055] Step 4:

[1056] If fraud is suspected, the device sends the text data to a server for further analysis. The server then further analyzes the text data and generates an appropriate response text, which includes an appropriate response to the fraudster. The input is the suspicious text data, and the output is the appropriate response text.

[1057] Step 5:

[1058] The server converts the generated response text into voice data using speech conversion technology and sends this voice data to the terminal. The terminal uses this voice data to immediately respond to the other party. The input is the response text data, and the output is the generated voice data.

[1059] Step 6:

[1060] At the same time, if fraud is suspected, the device will send a warning notification to family members or related parties. The warning notification is sent by means of email or other means and contains details of the suspected fraudulent call. The input is the fraud judgment result, and the output is a warning notification email to family members or related parties.

[1061] Step 7:

[1062] The communication device monitors text messages from messaging and social networking applications in real time and analyzes the text data. If a suspected fraudulent message is detected, it is sent to a server for further analysis. The input is the text message data collected in real time, and the output is a judgment result of whether the message is suspected to be fraudulent and an appropriate response.

[1063] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1064] This invention adds emotion recognition functionality to a system that detects and automatically responds to fraud targeting the elderly in real time through an application installed on a communication device. The invention includes server means for acquiring user account information and analyzing the user's voice to generate a voice model. It also includes means for converting call content into text in real time and analyzing the text to determine whether it is suspected of fraud. Suspicious call content is sent to the server, where it is analyzed in detail, and appropriate response text and voice responses are generated and sent to the terminal. This allows users to prevent fraud before it occurs. The system also has the ability to monitor text messages in messaging applications and social networking applications in real time and analyze suspected fraud.

[1065] Furthermore, this system incorporates an emotion engine that recognizes emotions from the user's voice and text messages during a call. Based on the user's emotions recognized by the emotion engine, the server adjusts the content of the response text. If the user's emotions are negative, such as stress or anxiety, the system has a means to automatically end the call.

[1066] Program processing overview and explanation

[1067] 1. User interface operation

[1068] First, the user installs and launches the app on their smartphone. They enter their user account information and provide data for identity verification. At this time, the user also records their voice to register. For example, elderly person A installs the app, enters account information such as their name and phone number, and records their voice.

[1069] 2. Server sends initial configuration data

[1070] The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice to generate an individual voice model. The generated voice model and initial setting data are sent to the device. For example, the server analyzes person A's voice to generate a voice model and sends that data to the smartphone.

[1071] 3. Monitor calls and messages

[1072] The device constantly monitors calls and messages in real time. When an incoming call is received, the call contents are converted into text using voice recognition technology. At the same time, text messages received via messaging applications and social networking applications are also monitored. For example, if a fraudster calls Person A, the device immediately converts the call contents into text and begins analyzing it.

[1073] 4. Analysis of call content and emotions

[1074] The device sends the text of the call and messages to an analysis engine, which detects suspicious keywords and phrases. It also uses an emotion engine to analyze the user's emotions. If it determines that fraud is suspected, it sends the analysis results and the call content to the server. For example, if the device detects phrases from the call content that are highly suspicious of fraud and determines that the user is feeling anxious, it sends the content to the server.

[1075] 5. Detailed analysis and response generation by the server

[1076] The server performs a detailed analysis based on the call content received from the device and the analysis results. If it determines that fraud is suspected using generative AI, it generates an appropriate response text. If the user's emotions are negative, it adjusts the content of the response text to match their emotions. A voice generation model is used to create a voice response based on the generated response text, and the voice data is sent to the device. For example, if the server analyzes the content of the call and determines that it contains fraud keywords such as "bank transfer," and the user is feeling anxious, it will generate a response including expressions such as "stay calm" and send it to the device.

[1077] 6. Speech generation and response

[1078] The device uses the received response text to generate a voice response to the call or message in question, instantly responding to the call or message. This response allows for effective countermeasures against scammers. If the user is extremely stressed, the call will be automatically terminated. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the device will convey this to the scammer. At the same time, if the user is under high stress, the call will be disconnected.

[1079] This series of processes allows users to quickly and effectively deal with even complex and sophisticated fraud methods. Furthermore, the system can monitor not only phone call content but also text messages on messaging and social networking applications in real time, providing similar countermeasures. By combining it with an emotion engine, the aim is to reduce the psychological burden on users and achieve safer and more secure communication.

[1080] The processing flow will be explained below.

[1081] Step 1: The user installs and launches the app on their smartphone. The user enters their account information and provides data for identity verification. At this time, the user also records their voice to register. For example, an elderly person named Mr. A installs the app, enters account information such as his name and phone number, and records his voice.

[1082] Step 2: The server receives the account information and voice data sent by the user and stores them in a database. Next, it analyzes the user's voice to generate an individual voice model. The generated voice model and initial setting data are sent to the device. For example, the server analyzes person A's voice to generate a voice model and sends that data to the smartphone.

[1083] Step 3: The device stores the initial setup data received from the server and prepares the app for operation. The voice model and various settings are activated on the device, and the app goes into standby mode. For example, the device receives the initial setup data and enables the analysis engine and speech generation engine.

[1084] Step 4: When the user receives a call, the device detects the incoming call and converts the call content into text using voice recognition technology. At the same time, text messages received via messaging and social networking applications are also monitored. For example, if a fraudster calls Person A, the device will immediately convert the call content into text and begin analyzing it.

[1085] Step 5: The device sends the converted call content and messages to an analysis engine, which detects suspicious keywords and phrases. It also uses an emotion engine to analyze the user's emotions. If it determines that fraud is suspected, it sends the analysis results and the call content to the server. For example, if the device detects phrases from the call content that are highly suspicious of fraud and determines that the user is feeling anxious, it sends the content to the server.

[1086] Step 6: The server performs a detailed analysis based on the call content received from the device and the analysis results. If it determines using generative AI that fraud is suspected, it generates an appropriate response text. If the user's emotions are negative, the content of the response text is adjusted to match the emotion. For example, if the server analyzes the call content and determines that it contains fraud keywords such as "transfer," and the user is feeling anxious, it will generate a response including expressions such as "remain calm" and send it to the device.

[1087] Step 7: The server creates a voice response using the speech generation model based on the generated response text and sends the voice data to the device. For example, the server generates a response such as "This call is suspected to be fraudulent. We will report this to the police" and sends it to the device as voice data.

[1088] Step 8: The device sends the received voice response to the other party and responds to the fraudster with a generated voice. This response can prevent fraud before it occurs. If the user is extremely stressed, the call can be automatically terminated. For example, if the device tells the fraudster in a generated voice, "This call is suspected to be fraud. We will report this to the police," and at the same time detects that the user is stressed, the call will be disconnected.

[1089] Step 9: After the call, the user can check the call history and analysis results within the app. This allows the user to confirm whether the call they received was safe. For example, if Person A checks the app after the call ends and learns from the analysis results that there was a suspicion of fraud.

[1090] Example 2

[1091] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1092] Fraud targeting the elderly is becoming more sophisticated every year, and is often difficult to prevent using conventional methods. Furthermore, real-time responses are required when a user is scammed, but existing technologies make it difficult to provide immediate responses. Furthermore, elderly users, who are more likely to fall victim to fraud, often experience anxiety and stress, and methods to alleviate this stress are also needed. Existing technologies lack systems that provide automated responses that take the user's emotional state into account, so there is an urgent need to develop a system that can reduce the psychological burden and prevent fraud quickly and effectively.

[1093] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1094] In this invention, the server includes means for acquiring user account information, means for analyzing the user's voice to generate a voice model, means for converting call content into text in real time, means for analyzing the converted text to determine whether a call is suspected of fraud, means for transmitting suspicious call content to the server, means for generating appropriate response text and creating a voice response using a generative AI model if a fraud is determined to be suspected, emotion recognition means for analyzing the user's emotions in real time, and means for automatically terminating the call based on the user's negative emotions. This makes it possible to detect calls or messages that are suspected of being fraudulent in real time and respond immediately. Furthermore, automatically terminating calls taking into account the user's emotional state significantly reduces psychological burden.

[1095] The "means for obtaining user account information" is a system component for obtaining personal information such as a user's name, phone number, email address, etc. through an application installed on a communication device.

[1096] The "server means for analyzing the user's voice and generating a voice model" is a system component on the server side that analyzes the voice data provided by the user and generates a voice model based on the voice characteristics.

[1097] "Means for converting call content into text in real time" refers to technology for converting voice data of incoming and outgoing calls into text data in real time.

[1098] The "means for analyzing the converted text to determine suspicion of fraud" is an analytical engine for analyzing the text content of the call and determining whether there is suspicion of fraud.

[1099] The "means for sending suspicious call content to a server" is a system component for sending the call content to a server if it is determined to be suspicious of fraud.

[1100] A "server means using a generative AI model" is a server-side system component that utilizes generative AI technology to generate appropriate response text and create a voice response based on that text.

[1101] "Emotion recognition means for analyzing user emotions in real time" is a technology for identifying and analyzing emotions in real time from a user's voice or text message.

[1102] The "means for automatically terminating a call based on a user's negative emotions" is a system component that automatically terminates a call when the user has negative emotions such as stress or anxiety.

[1103] This invention adds emotion recognition functionality to a system that detects and automatically responds to fraud targeting the elderly in real time. The invention includes a server means that acquires user account information through an application installed on a communication device, analyzes the user's voice, and generates a voice model.

[1104] Hardware and software used

[1105] Speech recognition technology: Uses the Google Cloud Speech-to-Text API to convert calls into text in real time.

[1106] Emotion Engine: Analyzes user emotions in real time using the Microsoft Azure Emotion API.

[1107] Database: Amazon RDS is used to store and manage user account information and voice models.

[1108] Generative AI: Uses OpenAI's GPT-3 to generate appropriate response text when suspected fraud is detected.

[1109] Call Management System: Uses Asterisk to capture real-time call data and perform text transcription.

[1110] Process Overview

[1111] First, the user installs and launches the app on their smartphone. They enter their user account information and provide data for identity verification. At this time, the user also records their voice to register. For example, elderly person A installs the app, enters account information such as their name and phone number, and records their voice.

[1112] The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice to generate an individual voice model. The generated voice model and initial setting data are sent to the device. For example, the server analyzes person A's voice to generate a voice model and sends that data to the smartphone.

[1113] The device constantly monitors calls and messages in real time. When an incoming call is received, the call contents are converted into text using voice recognition technology. At the same time, text messages received via messaging applications and social networking applications are also monitored. For example, if a fraudster calls Person A, the device immediately converts the call contents into text and begins analyzing it.

[1114] The device sends the text of the call and messages to an analysis engine, which detects suspicious keywords and phrases. It also uses an emotion engine to analyze the user's emotions. If it determines that fraud is suspected, it sends the analysis results and the call content to the server. For example, if the device detects phrases from the call content that are highly suspicious of fraud and determines that the user is feeling anxious, it sends the content to the server.

[1115] The server performs a detailed analysis based on the call content received from the device and the analysis results. If it determines that fraud is suspected using generative AI, it generates an appropriate response text. If the user's emotions are negative, it adjusts the content of the response text to match their emotions. A voice generation model is used to create a voice response based on the generated response text, and the voice data is sent to the device. For example, if the server analyzes the content of the call and determines that it contains fraud keywords such as "bank transfer" and the user is feeling anxious, it will generate a response including expressions such as "stay calm" and send it to the device.

[1116] The device uses the received response text to generate a voice response to the call or message in question, instantly responding to the call or message. This response allows for effective countermeasures against scammers. If the user is extremely stressed, the call will be automatically terminated. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the device will convey this to the scammer. At the same time, if the user is under high stress, the call will be disconnected.

[1117] Specific prompt sentence examples and output examples

[1118] Prompt statement:

[1119] User says: "This is from your bank. We're having trouble with your account..."

[1120] Analysis results:

[1121] Fraud suspicion level: High

[1122] Sentiment analysis: Determines that the user's emotion is "anxiety"

[1123] Generates the following response:

[1124] This call is suspected to be fraudulent and I will be reporting the details to the police.

[1125] Example output (response after speech generation):

[1126] This call is suspected to be a scam. I will report the details to the police. Please remain calm.

[1127] This invention allows users to prevent fraud and alleviate psychological stress through emotion recognition. It provides a system that monitors calls and analyzes emotions in real time, allowing for quick and effective response.

[1128] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1129] Step 1:

[1130] A user installs and launches an app on their smartphone, which includes downloading and installing it from the app store and then tapping the app's icon to launch it.

[1131] Input: User terminal operation.

[1132] Output: The app is launched.

[1133] Step 2:

[1134] Users enter their user account information within the app and provide data to verify their identity, such as their name, phone number, and email address, and record and register their own voice to create a voice model.

[1135] Input: Name, phone number, email address, recorded voice data.

[1136] Output: Completed user account information and audio data.

[1137] Step 3:

[1138] The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice using speech recognition technology (Google Cloud Speech-to-Text API) to generate a voice model. The generated voice model and initial setting data are sent to the device.

[1139] Input: User account information, recorded voice data.

[1140] Output: User information stored in the database. Generated speech model and initial configuration data.

[1141] Step 4:

[1142] The device goes into constant monitoring mode and monitors calls and messages in real time. When an incoming call is received, the conversation is captured and converted into text using the Google Cloud Speech-to-Text API. Similarly, text messages received via messaging and social networking applications are also monitored.

[1143] Input: Calls and text messages.

[1144] Output: Transcript of calls and monitoring records.

[1145] Step 5:

[1146] The device sends the transcribed conversations and messages to an analytics engine that detects suspicious keywords and phrases and analyzes the user's emotions using the Microsoft Azure Emotion API.

[1147] Input: Texted calls and messages.

[1148] Output: Detected suspicious keywords and phrases, analyzed sentiment data.

[1149] Step 6:

[1150] The device sends the analysis results to the server. The server performs a detailed analysis based on the received analysis results and the content of the call. If it determines that fraud is suspected using generative AI (GPT-3), it generates an appropriate response text. If the user's sentiment is judged to be negative, the content of the response text is adjusted.

[1151] Input: Analysis results, emotion data.

[1152] Output: Detailed analysis results, generated response text.

[1153] Step 7:

[1154] The server creates a voice response using a speech generation model based on the generated response text, and sends the generated voice data to the terminal.

[1155] Input: The response text.

[1156] Output: The generated audio data.

[1157] Step 8:

[1158] The device responds to calls and messages in real time using the voice data it receives, and automatically ends calls if the user is feeling extremely stressed.

[1159] Input: The generated audio data.

[1160] Output: Answered voice message, call ended.

[1161] This process effectively protects users from fraud targeting the elderly, and real-time monitoring and sentiment analysis enable swift and appropriate responses.

[1162] (Application example 2)

[1163] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1164] Fraud targeting the elderly is increasing year by year, and the methods are becoming more sophisticated and ingenious. In particular, fraud perpetrated via telephone or messaging services has the potential to cause significant damage if not dealt with promptly. Furthermore, victims often experience significant emotional stress and anxiety when they encounter fraud. However, current fraud prevention systems lack the ability to address these emotional needs. Therefore, the present invention aims to provide a more effective and secure fraud prevention system for the elderly by combining fraud detection and emotion recognition.

[1165] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server is installed in the communication device and includes means for acquiring user account information, means for analyzing the user's voice to generate a voice model, and means for converting the content of the call into text in real time. This includes means for generating appropriate response text and voice response while taking the user's emotional state into consideration when fraud is suspected, and means for adjusting the content of the response text if the user's emotions are negative and automatically terminating the call if the user's stress level is high. This not only detects fraudulent acts but also enables fraud prevention measures that reduce the mental burden on elderly people and allow them to use the service with peace of mind.

[1166] "Communication device" refers to a device that transmits and receives information using telecommunications technology, including smartphones and tablets.

[1167] "User account information" refers to data containing a user's personal information and authentication information, such as name, phone number, and password.

[1168] A "voice model" is a computer model generated by analyzing the vocal characteristics of a particular user and used to reproduce the user's voice patterns.

[1169] "Server means" refers to a collection of server devices and server programs for providing specific functions and services.

[1170] "Means for converting to text" refers to the technology or device that converts voice data into text data. A voice recognition engine is an example of this.

[1171] "Means for analyzing text to determine suspected fraud" refers to technologies or programs that analyze converted character data and assess the likelihood of fraud.

[1172] "Emotion engine" refers to algorithms or software that identify a user's emotional state from speech or text.

[1173] "Negative emotions" refers to negative emotional states such as anxiety, stress, anger, and sadness.

[1174] "Means for generating appropriate response text" refers to technology or a program that creates an appropriate message for the user or the other party based on the analysis results.

[1175] "Voice response" refers to generating voice data that reads out text data.

[1176] "Means for automatically terminating a call" refers to a mechanism or technology that forcibly terminates a call when certain conditions are met.

[1177] This invention provides a system for detecting and automatically combating fraud targeting seniors in real time through an application installed on a communication device. The invention includes the following main components:

[1178] User interface operations

[1179] First, the user installs and launches the application on their smartphone. They must enter their account information and provide data for identity verification. At this point, the user registers by recording their voice. For example, an elderly person named Mr. A installs the app, enters account information such as his name and phone number, and records his voice.

[1180] Server sends initial configuration data

[1181] The server receives the account information and voice data sent by the user and stores them in a database. It then analyzes the user's voice to generate an individual voice model. The generated voice model and initial setting data are sent to the device. For example, the server analyzes person A's voice to generate a voice model and sends that data to the smartphone.

[1182] Call and message monitoring

[1183] The device constantly monitors calls and messages in real time. When an incoming call is received, the contents of the call are converted into text using voice recognition technology. At the same time, text messages received via messaging applications and social networking applications are also monitored. For example, if Person A receives a call from a fraudster, the device immediately converts the contents of the call into text and begins analysis.

[1184] Call content and emotion analysis

[1185] After the call contents are converted into text, they are sent to an analysis engine, which detects suspicious keywords and phrases. The emotion engine also analyzes the user's emotions. If fraud is suspected, the analysis results and the contents of the call are sent to the server. For example, if the device detects phrases from the call contents that are highly suspicious of fraud and determines that the user is feeling anxious, the content is sent to the server.

[1186] Detailed analysis and response generation by the server

[1187] The server performs a detailed analysis based on the call content received from the device and the analysis results. If it determines that fraud is suspected using a generative AI model, it generates an appropriate response text. If the user's emotions are negative, it adjusts the content of the response text to match the emotion. Based on the generated response text, it creates a voice response using a voice generation model, and sends the voice data to the device. For example, if the server analyzes the content of the call and determines that it contains fraud keywords such as "bank transfer," and the user is feeling anxious, it generates a response including the expression "stay calm" and sends it to the device.

[1188] Speech generation and response

[1189] The device uses the received response text to generate a voice response to the call or message in question, instantly responding to the call or message. This response allows for effective countermeasures against scammers. If the user is extremely stressed, the call will be automatically terminated. For example, if the generated voice response is "This call is suspected to be fraudulent. We will report this to the police," the device will convey this to the scammer. At the same time, if the user is under high stress, the call will be disconnected.

[1190] Examples and prompts

[1191] For example, consider the case where elderly person A receives a phone call that is suspected to be a scam. Speech recognition recognizes A's statement that "I need to transfer money," and emotion analysis determines that the stress score is high (e.g., 0.9). The fraud detection model detects the suspicion of fraud from the keyword "transfer," and the response generation model creates an appropriate response such as, "This call may be a scam. Please remain calm and do not respond to the caller's requests. The call will end."

[1192] Example prompt for a generative AI model:

[1193] We provide the call content so you can determine the emotional state: "I need to transfer money."

[1194] This invention will enable elderly people to prevent themselves from falling victim to fraud, reduce their mental burden, and live their daily lives with peace of mind.

[1195] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1196] Step 1:

[1197] User interface operations:

[1198] A user installs and launches an application on a communication device. The user enters account information (such as name and phone number) and records their voice to register. At this time, the communication device collects the entered account information and voice data and sends them to a server. The input data is the user's personal information and voice data, and the output is initial setting data sent to the server.

[1199] Step 2:

[1200] Server sends initialization data:

[1201] The server receives the account information and voice data sent by the user and stores them in a database. The server then analyzes the received voice data to generate a voice model. This voice model reproduces the user's voice characteristics. The generated voice model and initial setting data are sent to the communication device. The data processing required in this process is voice analysis, and the output is a voice model and initial setting data.

[1202] Step 3:

[1203] Call and message monitoring:

[1204] The communication device monitors all calls and messaging application interactions in real time. When an incoming call is received, the call content is converted into text using voice recognition technology. It also monitors text messages from messaging applications and social networking applications. In this monitoring process, the input is voice data and text data, and the output is the converted call content.

[1205] Step 4:

[1206] Call content and sentiment analysis:

[1207] The communication device sends the text of the call and messages to an analysis engine to detect keywords and phrases that may indicate fraud. It also uses an emotion engine to analyze the user's emotional state. If it determines that fraud is suspected, it sends that information to a server. In this step, the input is the text of the call, and the output is the analysis result and the emotional state.

[1208] Step 5:

[1209] Detailed analysis and response generation by the server:

[1210] The server performs a detailed analysis based on the call content and analysis results sent from the communication device. If suspicion of fraud is confirmed, the server uses a generative AI model to generate an appropriate response text. It also adjusts the response content based on the user's emotional state. It creates a voice response using a speech generation model based on the generated response text, and sends the voice data to the communication device. The inputs to this step are the analysis results and the emotional state, and the output is voice response data.

[1211] Step 6:

[1212] Speech generation and response:

[1213] The communication device generates a voice based on the response text received from the server and immediately responds to the caller or message recipient. It also automatically ends the call if the user is extremely stressed. In this process, the input is the voice response data and the output is the end of the call or message.

[1214] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1215] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1216] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1217] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1218] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1219] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1220] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1221] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1222] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1223] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1224] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1225] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1226] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1227] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1228] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1229] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1230] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1231] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1232] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1233] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1234] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1235] The following is further disclosed regarding the above embodiment.

[1236] (Claim 1)

[1237] means installed on the communication device for obtaining user account information;

[1238] a server means for analyzing a user's voice and generating a voice model;

[1239] A means of converting phone calls into text in real time;

[1240] means for analyzing the converted text to determine suspected fraud; and

[1241] a means for transmitting suspicious call content to a server;

[1242] a server means for generating an appropriate response text and creating a voice response when it is determined that fraud is suspected;

[1243] means for transmitting the generated voice response to the other party;

[1244] A system including:

[1245] (Claim 2)

[1246] 10. The system of claim 1, further comprising means for alerting a user to a suspected fraudulent call.

[1247] (Claim 3)

[1248] 10. The system of claim 1, further comprising means for monitoring text messages in messaging and social networking applications in real time and analyzing the messages for suspected fraud.

[1249] "Example 1"

[1250] (Claim 1)

[1251] means installed on the communication device for obtaining user account information;

[1252] a server means for analyzing a user's voice and generating a voice model;

[1253] A means of converting phone calls into text in real time;

[1254] means for analyzing the converted text to determine suspected fraud; and

[1255] a means for transmitting suspicious call content to a server;

[1256] a server means for generating an appropriate response text and creating a voice response when it is determined that fraud is suspected;

[1257] means for transmitting the generated voice response to the other party;

[1258] means for notifying the user of the suspected fraud when the call is terminated;

[1259] A system including:

[1260] (Claim 2)

[1261] 10. The system of claim 1, further comprising means for alerting a user to a suspected fraudulent call.

[1262] (Claim 3)

[1263] 10. The system of claim 1, further comprising means for monitoring text messages of messaging applications and social network service applications in real time and analyzing the messages for suspected fraud.

[1264] "Application Example 1"

[1265] (Claim 1)

[1266] means installed on the communication device for obtaining user account information;

[1267] a server means for analyzing a user's voice and generating a voice model;

[1268] A means of converting phone calls into text in real time;

[1269] means for analyzing the converted text to determine suspected fraud; and

[1270] a means for transmitting suspicious call content to a server;

[1271] a server means for generating an appropriate response text and creating a voice response when it is determined that fraud is suspected;

[1272] means for transmitting the generated voice response to the other party;

[1273] A means of issuing warning notices to family members or contacts in the event of suspected fraud;

[1274] A system including:

[1275] (Claim 2)

[1276] 10. The system of claim 1, further comprising means for alerting a user to a suspected fraudulent call.

[1277] (Claim 3)

[1278] 10. The system of claim 1, further comprising means for monitoring text messages in messaging and social networking applications in real time and analyzing the messages for suspected fraud.

[1279] "Example 2: Combining Emotion Engines"

[1280] (Claim 1)

[1281] means installed on the communication device for obtaining user account information;

[1282] a server means for analyzing a user's voice and generating a voice model;

[1283] A means of converting phone calls into text in real time;

[1284] means for analyzing the converted text to determine suspected fraud; and

[1285] a means for transmitting suspicious call content to a server;

[1286] A server means using a generative AI model to generate appropriate response text and create a voice response when fraud is suspected;

[1287] means for transmitting the generated voice response to the other party;

[1288] emotion recognition means for analyzing the user's emotions in real time;

[1289] means for automatically terminating a call based on a user's negative sentiment;

[1290] A system including:

[1291] (Claim 2)

[1292] 10. The system of claim 1, further comprising means for alerting a user to a suspected fraudulent call.

[1293] (Claim 3)

[1294] 10. The system of claim 1, further comprising means for monitoring text messages in messaging and social networking applications in real time and analyzing the messages for suspected fraud.

[1295] "Application example 2 when combining emotion engines"

[1296] (Claim 1)

[1297] means installed on the communication device for obtaining user account information;

[1298] a server means for analyzing a user's voice and generating a voice model;

[1299] A means of converting phone calls into text in real time;

[1300] means for analyzing the converted text to determine suspected fraud; and

[1301] a means for transmitting suspicious call content to a server;

[1302] a server means for generating an appropriate response text and creating a voice response when it is determined that fraud is suspected;

[1303] means for transmitting the generated voice response to the other party;

[1304] means for analyzing the emotional state of the user using an emotion engine that recognizes the user's emotions;

[1305] means for adjusting the content of an appropriate response text if the user's sentiment is negative;

[1306] A means for automatically ending a call when the user is under high stress;

[1307] A system including:

[1308] (Claim 2)

[1309] 10. The system of claim 1, further comprising means for alerting a user to a suspected fraudulent call.

[1310] (Claim 3)

[1311] 10. The system of claim 1, further comprising means for monitoring text messages in messaging and social networking applications in real time and analyzing the messages for suspected fraud. [Explanation of symbols]

[1312] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means installed on the communication device for obtaining user account information; a server means for analyzing a user's voice and generating a voice model; A means of converting phone calls into text in real time; means for analyzing the converted text to determine suspected fraud; and a means for transmitting suspicious call content to a server; a server means for generating an appropriate response text and creating a voice response when it is determined that fraud is suspected; means for transmitting the generated voice response to the other party; A system including:

2. 10. The system of claim 1, further comprising means for alerting a user to a suspected fraudulent call.

3. 10. The system of claim 1, further comprising means for monitoring text messages in messaging and social networking applications in real time and analyzing the messages for suspected fraud.

Images