System

Abstract

A system is provided.SOLUTION: A system comprising: means for capturing and converting a user's voice to text in real-time; means for analyzing the text using a generative AI model to assess a likelihood of fraud; and means for sending an alert to a pre-registered contact if a likelihood of fraud is detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, fraudulent billing scams have become increasingly sophisticated, with many people, especially the elderly, falling victim to them. Existing countermeasures are limited to warning users in advance and are only effective when a fraud is actually in progress. Therefore, there is a need for an effective system that can detect potentially fraudulent conversations in real time and respond quickly. The present invention aims to solve these problems and provide an effective means for preventing fraud damage before it occurs. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for capturing a user's voice in real time and converting it into text, a means for analyzing the text using a generative AI model to assess the likelihood of fraud, and a means for sending an alert to pre-registered contacts if a possible fraud is detected. This system enables rapid and accurate detection of fraudulent billing scams and prevents fraud damage. The pre-registered contacts also include family members and the police, enabling a rapid response. Furthermore, the generative AI model performs analysis by comparing the results with a database of past fraud cases, achieving highly accurate fraud detection.

[0006] "User" refers to an individual using the system and / or subject to fraud detection.

[0007] "Means for capturing voice in real time" refers to hardware such as a microphone that constantly collects the user's voice and instantly captures it as data, as well as software that controls it.

[0008] "Text conversion means" means speech recognition technology and software used to convert captured audio data into a format that can be parsed and displayed as text.

[0009] A "generative AI model" refers to an artificial intelligence algorithm model that generates new insights and analytical results based on past data.

[0010] "Means of analyzing" refers to the process of analyzing acquired text data using a generative AI model to detect specific patterns or anomalies.

[0011] "Possible fraud" refers to an event that is determined to be highly likely to be fraud based on the analysis results.

[0012] "Means of evaluation" refers to the mechanism by which the results of the analysis are used to determine the presence or likelihood of fraud.

[0013] "Means for sending alerts" refers to a system for automatically sending warning messages to pre-registered contacts when potential fraud is detected.

[0014] "Pre-registered contacts" refers to contact information for family members, police, or other individuals whom the user would like to be notified of if fraud is detected.

[0015] "System" refers to an integrated set of hardware and software that operates in a comprehensive combination of the above means.

[0016] "Database" refers to a collection of information that collects and manages past fraud cases and related information. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] overview

[0039] This invention is a system that constantly monitors users' conversations to detect fraudulent billing in real time and sends alerts to family members and the police if there is a possibility of fraud. This system includes an analysis function using a generative AI model, and can make highly accurate judgments by comparing the results with a database of past fraud cases.

[0040] Configuring Users

[0041] Users install a dedicated app and perform the initial setup, registering contact information such as family members and the police. This allows the device to be set up so that alerts are sent to the registered contacts if fraud is detected.

[0042] Audio Capture and Conversion

[0043] The device is equipped with a microphone for voice capture, which records the user's conversation in real time. This voice data is converted into text using voice recognition technology. For example, if a user says, "Tell me the PIN number for my cash card," the voice data is instantly converted into text data.

[0044] Analysis and Evaluation

[0045] The converted text is sent from the device to a server, which uses a generative AI model to analyze the text and assess its likelihood of fraud. The generative AI model then compares the text with a database of past fraud cases to determine whether it indicates fraud.

[0046] Sending alerts

[0047] If a potential fraudulent activity is detected, the server will send an alert to registered contacts. The alert will include the content of the potentially fraudulent conversation and a message urging immediate action. For example, a notification might say, "A potentially fraudulent conversation has been detected. Please check immediately."

[0048] Specific examples

[0049] 1. User comments

[0050] User: "What's my cash card PIN?"

[0051] The device captures this audio and converts it into text.

[0052] Converted text: "What is your cash card PIN?"

[0053] 2. Server Analysis

[0054] The text is sent to a server and analyzed by a generative AI model.

[0055] The model compares the data with past fraud cases and rates the likelihood of fraud highly.

[0056] 3. Sending alerts

[0057] The server detects potential fraud and sends alerts to registered family members and the police.

[0058] Alert: "A potentially fraudulent conversation has been detected. Please investigate immediately."

[0059] merit

[0060] This system enables real-time detection of fraudulent billing scams and rapid response, providing effective crime prevention measures, particularly for individuals at high risk of fraud, such as the elderly. Furthermore, the use of generative AI models can improve the accuracy of fraud detection and reduce false positives.

[0061] The processing flow will be explained below.

[0062] Step 1:

[0063] Users install the app, launch it, and perform the initial setup, which includes registering contact information for family members, police, etc. This sets up where alerts will be sent if fraud is detected.

[0064] Step 2:

[0065] The terminal uses a microphone for voice capture to record the user's conversation in real time. This voice data is immediately sent to a voice recognition system by a processing unit in the terminal and converted into text format. For example, if a user says, "Tell me the PIN number for my cash card," this voice is converted into text data, "Tell me the PIN number for my cash card."

[0066] Step 3:

[0067] The converted text data is sent from the device to a server, which receives the text data and begins analyzing it using a generative AI model.

[0068] Step 4:

[0069] The server analyzes the received text data using a generative AI model to assess the likelihood of fraudulent activity, which then matches the text with past fraud cases stored in a database and runs algorithms to detect signs of fraud.

[0070] Step 5:

[0071] If the generative AI model's analysis determines that fraud is likely, the server generates an alert, which includes a description of the potential fraud and the conversation.

[0072] Step 6:

[0073] The server generates an alert and sends it to pre-registered contacts, including the user's family and the police, with a message urging them to immediately investigate the fraudulent activity, such as "A potentially fraudulent conversation has been detected. Please investigate immediately."

[0074] Step 7:

[0075] After an alert is sent, the server logs the results, including the fraudulent activity detected, the alert content, and the destination information, for later analysis and auditing.

[0076] Step 8:

[0077] Users who receive the alert can work with their family or the police to take action to prevent the fraud from progressing, for example by taking specific measures such as verifying personal information with callers before providing it.

[0078] This process flow protects users from fraudulent billing and allows for a quick response.

[0079] Example 1

[0080] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0081] Fraudulent activities, such as fictitious billing scams, have a serious impact, especially on vulnerable individuals such as the elderly. To address this issue, a system that can detect fraud in real time and respond quickly is needed. However, conventional methods have low accuracy in detecting fraud, and are prone to false positives and delays. Therefore, a method is needed to detect fraud in real time with high accuracy and respond quickly.

[0082] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0083] In this invention, the server includes a means for capturing a user's voice in real time and converting it into text, a means for analyzing the text using a generative AI model to evaluate the likelihood of fraud, a means for sending an alert to pre-registered contacts if a likelihood of fraud is detected, a means for comparing the text with a database of past fraud cases, and a means for sending the text to the server using encrypted communication, thereby enabling fraud to be detected with high accuracy in real time and alerting family members and the police quickly.

[0084] "Means for capturing and converting a user's voice into text in real time" refers to a device or software that instantly records the voice data spoken by a user and converts it into text data using voice recognition technology.

[0085] "Analyzing said text using a generative AI model to assess the likelihood of fraud" refers to technology that uses machine learning models to analyze text data and determine the likelihood of fraud based on specific patterns or characteristics of fraud.

[0086] "Means for sending alerts to pre-defined contacts when potential fraud is detected" refers to a method or device for sending a warning message to pre-defined family or law enforcement contacts when signs of fraud are detected.

[0087] "Means for comparing the text with a database of past fraud cases" refers to technology for determining fraudulent activity by referring to a database that records past fraud cases and comparing and matching the text data to be analyzed.

[0088] "Means for transmitting the text to the server using encrypted communication" refers to a method that uses data encryption technology to securely transmit text data to the server.

[0089] overview

[0090] This invention is a system that detects fraudulent billing in real time by constantly monitoring the user's voice and sends an alert to family members and the police if there is a possibility of fraud. This system performs analysis using a generative AI model and compares the results with a database of past fraud cases, enabling highly accurate judgments.

[0091] System Configuration

[0092] This system is realized mainly using the following hardware and software.

[0093] Hardware

[0094] Device: A smartphone or tablet with a microphone for audio capture.

[0095] Server: A high-performance computer for running generative AI models.

[0096] software

[0097] Generative AI model: A machine learning model that analyzes fraudulent activity.

[0098] Speech recognition software: For example, using the Google Speech-to-Text API.

[0099] Database: A database containing past fraud cases.

[0100] Specific processing of the program

[0101] User Preferences

[0102] Users install a dedicated app on their device and register contact information for family members and the police, so that if fraud is detected, an alert is automatically sent to the designated contacts.

[0103] Audio capture and transcription

[0104] The device's built-in microphone constantly records the user's speech and converts this voice data into text data in real time. For example, if a user says, "Tell me my cash card PIN," the speech is converted into text using voice recognition software.

[0105] Sending and analyzing text data

[0106] The converted text data is sent to a server using encrypted communications. The server then analyzes the received text data using a generative AI model to assess the likelihood of fraud. This analysis includes a process of comparing it with a database of past fraud cases.

[0107] Sending alerts

[0108] If a fraudulent activity is detected, the server will send an alert to registered family members or the police. The alert will include the details of the potentially fraudulent conversation and a message urging immediate action. For example, a notification such as "A potentially fraudulent conversation has been detected. Please check immediately."

[0109] Examples and prompts

[0110] Specific examples

[0111] 1. User utterance: The user says, "Tell me the PIN number for my cash card."

[0112] What the device does: It captures this audio and converts it to text using speech recognition software.

[0113] Converted text: "What is your cash card PIN?"

[0114] 2. Server analysis: The converted text is sent to the server and analyzed using the generative AI model.

[0115] Analysis results: The generative AI model compares the data with a database of past fraud cases and evaluates the likelihood of fraudulent activity.

[0116] 3. Send alert: The server determines that there is a high possibility of fraud and sends an alert to pre-registered family members and the police.

[0117] Alert: "A potentially fraudulent conversation has been detected. Please investigate immediately."

[0118] Prompt Sentence Examples

[0119] To perform analysis, enter the following prompt into the generative AI model:

[0120] "Judge whether the following text is likely to be fraudulent: 'Tell me your bank card PIN'"

[0121] As described above, the present invention is a system that aims to analyze user conversations in real time, detect fraudulent acts with high accuracy, and respond quickly.

[0122] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0123] Step 1: Initial User Setup

[0124] Users install a dedicated app on their device. When they launch the app for the first time, they register contact information for family members and the police, which allows the app to send prompt notifications when fraud is detected. The input is the contact information entered by the user, and the output is that it is saved on the server.

[0125] Input: Family and police contact information

[0126] Output: Contact information is saved on the server

[0127] Specific operation: The user opens the app, enters the required information on the contact registration screen, and presses the save button.

[0128] Step 2: Capture audio

[0129] The device uses a built-in microphone to capture the user's speech in real time. The captured voice data is temporarily stored on the device. The input is the user's voice and the output is the captured voice data.

[0130] Input: User spoken words

[0131] Output: Captured audio data

[0132] What it does: The device runs the app in the background, and the microphone constantly captures audio and stores it in temporary storage.

[0133] Step 3: Convert audio data to text

[0134] The device converts the captured voice data into text data in real time using voice recognition software (e.g., Google Speech-to-Text API). The input is the captured voice data, and the output is text data.

[0135] Input: Captured audio data

[0136] Output: Converted text data

[0137] Specific operation: The saved voice data is converted into text data by voice recognition software.

[0138] Step 4: Sending text data

[0139] The terminal sends the converted text data to the server using encrypted communication. The input is the converted text data, and the output is the data sent to the server.

[0140] Input: Converted text data

[0141] Output: Text data is sent to the server

[0142] Specific operation: Once text data is generated, it is automatically sent to the server, and SSL / TLS encrypted communication is used.

[0143] Step 5: Analyzing the text data

[0144] The server analyzes the received text data using a generative AI model to assess the likelihood of fraud. The generative AI model makes its judgment based on previously learned fraud patterns. The input is the text data sent to the server, and the output is the assessment result of fraud.

[0145] Input: Text data sent to the server

[0146] Output: Fraud assessment results

[0147] Specific operation: The server inputs the received text data into a generative AI model, which then analyzes the data by comparing it with a database of past fraud cases.

[0148] Step 6: Generate fraud findings

[0149] The server receives the analysis results from the generative AI model and evaluates whether the likelihood of fraud is high or low. If it is judged to be high, it is considered fraud. The input is the analyzed text data, and the output is the fraud judgment result.

[0150] Input: Parsed text data

[0151] Output: Fraud detection result

[0152] Specific operation: The server receives the analysis evaluation results and identifies the possibility of fraud using a number or tag.

[0153] Step 7: Sending an alert

[0154] If the server determines that there is a high possibility of fraud, it will send an alert to registered family members or the police. The input is the fraud judgment result and pre-registered contact information, and the output is an alert message.

[0155] Input: Fraud detection result, registered contact information

[0156] Output: Alert message

[0157] What happens: The server generates an alert message and sends it via SMS, email, or in-app notification. Notification content: "A potentially fraudulent conversation has been detected. Please check immediately."

[0158] (Application example 1)

[0159] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0160] In recent years, there has been a sharp increase in fraudulent activities such as fictitious billing scams and bank transfer scams, with elderly people and those in socially vulnerable positions being particularly targeted. Conventional fraud prevention measures are limited to crime prevention education and warnings, and there are limitations to detecting fraud in real time and responding quickly, making it difficult to quickly prevent fraud damage, so a solution is needed.

[0161] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0162] In this invention, the server includes means for capturing a user's voice in real time and converting it into text, means for analyzing the text using a generative AI model to evaluate the possibility of fraud, means for sending an alert to pre-registered contacts when a possibility of fraud is detected, means for being installed on the smartphone and monitoring the user's calls in real time, means for recording the user's conversations and analyzing the content using a generative AI model, means for comparing the content with a database of past fraud cases to evaluate signs of fraud, and communication means for instantly sending the alert, thereby enabling fraud detection in real time and rapid response.

[0163] "Real-time voice capture" refers to technology that instantly records the voice spoken by the user and acquires the voice data.

[0164] "Convert to text" refers to the process of converting audio data into written information.

[0165] A "generative AI model" is an artificial intelligence algorithm or machine learning model that has been trained for a specific task.

[0166] "Analysis" is the process of evaluating and understanding data to understand its meaning and content.

[0167] "Fraudulent Potential" refers to an indicator or assessment of the likelihood that an action or statement may be fraudulent.

[0168] "Alert" means a notice or warning of an emergency or caution.

[0169] "Pre-registered contacts" refers to the phone numbers and email addresses that the user has set up in advance to receive notifications.

[0170] A "smartphone" refers to a mobile information terminal that has communication capabilities, internet connectivity, and can run applications.

[0171] "Real-time call monitoring" refers to the process of instantly monitoring and tracking user calls and communications.

[0172] "Recording audio" refers to the act of saving a user's speech or conversation on a recording medium.

[0173] "Content analysis" refers to the process of examining recorded data to understand its meaning and intent.

[0174] A "database of past fraud cases" refers to a collection of information that has accumulated data on fraud cases that have occurred in the past.

[0175] "Assessing for fraudulent indicators" refers to the process of determining the likelihood that a particular action or statement may be fraudulent and drawing a consequence.

[0176] "Communication means" refers to the technology and infrastructure for transmitting information, including the Internet and mobile communications.

[0177] System configuration

[0178] The present invention is implemented as a security application installed on a smartphone, which monitors users' calls in real time and performs a series of processes to detect fraudulent activity.

[0179] Hardware and software used

[0180] Smartphone: microphone, internet connection

[0181] Speech recognition software: Google Cloud Speech-to-Text API

[0182] Generative AI model: OpenAI GPT-4

[0183] Database: Database of past fraud cases

[0184] Communication Method: Uses Twilio API to send alerts via SMS and phone

[0185] Data processing and calculation

[0186] Audio Capture

[0187] The user's call is recorded in real time through the smartphone's microphone, and the audio data is captured in high quality and is used directly for the next step, which is to convert it into text.

[0188] Speech-to-text

[0189] The recorded voice data is converted into text data in real time using the Google Cloud Speech-to-Text API, so the words spoken by the user are recorded as text and can be used directly in subsequent analysis processes.

[0190] Text analytics

[0191] The generative AI model (OpenAI GPT-4) uses text data to analyze the content of utterances, using a pre-filled prompt, "Is this likely to be fraud?", to accurately assess the likelihood of fraud.

[0192] Comparison with a database of past fraud cases

[0193] The analyzed text data is compared with a database of past fraud cases, which contains a large number of past fraud methods and cases, and based on this, the system evaluates signs of fraud with high accuracy.

[0194] Sending alerts

[0195] If a fraudulent activity is deemed likely, an alert will be sent immediately to the user's pre-registered contacts (family, police, etc.) using the Twilio API. The alert will include the contents of the detected conversation and a message urging immediate action.

[0196] Specific examples

[0197] 1. Audio Capture

[0198] A user says on the phone, "Tell me the PIN number for my cash card."

[0199] The smartphone's microphone records this audio.

[0200] 2. Text Conversion

[0201] The recorded voice is converted into "Please tell me my cash card PIN number" via the Google Cloud Speech-to-Text API.

[0202] 3. Text Analysis

[0203] A generative AI model (OpenAI GPT-4) analyzes the text and evaluates it against the prompt, "Is this a possible scam?"

[0204] Evaluation result: "High probability of fraud"

[0205] 4. Sending alerts

[0206] Alerts are instantly sent to the user's family and police using the Twilio API.

[0207] Alert: "A potentially fraudulent conversation has been detected. Please investigate immediately."

[0208] Prompt Sentence Examples

[0209] text

[0210] Fraud prevention system:

[0211] Text generated from user's call data: "What is my cash card PIN?"

[0212] Prompt: "Could this be a scam?"

[0213] In this way, the present invention is able to detect fraudulent activity in real time and respond quickly.

[0214] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0215] Step 1:

[0216] A user initiates a call through a smartphone. The user's speech is captured in real time by the smartphone's microphone and collected as audio data. The input is the user's speech, and the output is the recorded audio data.

[0217] Step 2:

[0218] The device sends the captured audio data to the Google Cloud Speech-to-Text API, where it is analyzed and converted to text. The input is the original audio data, and the output is the converted text data.

[0219] Step 3:

[0220] The device sends the converted text data to a generative AI model (OpenAI GPT-4), which analyzes the text data using a pre-defined prompt: "Is this likely to be fraud?" The input is the text data and the prompt, and the output is an assessment of the likelihood of fraud.

[0221] Step 4:

[0222] The server receives the evaluation results from the generative AI model and compares them with a database of past fraud cases, where patterns and features that match fraudulent activity are further evaluated. The input is the text analysis results and database information, and the output is the final evaluation result.

[0223] Step 5:

[0224] If fraud is determined to be likely, the server uses the Twilio API to send an alert to pre-registered contacts, where the input is the evaluation result and registered contact information, and the output is the sent alert message.

[0225] Step 6:

[0226] Finally, the user's family or the police receive the alert and quickly confirm and take necessary action. In this step, the input is the alert message and the output is the confirmation of the message and the corresponding action.

[0227] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0228] overview

[0229] This invention is a system that constantly monitors users' conversations and emotions to detect fraudulent billing scams in real time, and sends alerts to family members and the police if there is a possibility of fraud. This system uses a generative AI model and an emotion engine to predict and detect fraud damage with high accuracy.

[0230] Configuring Users

[0231] Users install the app and perform the initial setup, which includes registering contact information for family members, police, etc. This sets up where alerts will be sent if fraud is detected.

[0232] Audio Capture and Conversion

[0233] The device is equipped with a microphone for voice capture, which records the user's conversation in real time. This voice data is converted into text using voice recognition technology. For example, if a user says, "Tell me the PIN number for my cash card," this voice is converted into text data: "Tell me the PIN number for my cash card."

[0234] emotion recognition

[0235] The converted voice data is analyzed by the device's emotion engine, which detects the user's emotional state from the voice data. This analysis evaluates whether the user is feeling stressed, relaxed, anxious, etc.

[0236] Analysis and Evaluation

[0237] The converted text data and the results of the sentiment analysis are sent from the device to a server. The server uses a generative AI model to analyze the text data and evaluate the likelihood of fraud. The generative AI model compares the text with past fraud cases stored in a database to detect signs of fraud. The user's emotional state also influences the evaluation.

[0238] Sending alerts

[0239] If the generative AI model's analysis determines that fraud is highly likely, the server generates an alert. This alert includes information about the potential fraud, the content of the conversation, and the user's emotional state. The alert is sent to pre-registered contacts (family members or police). If the user's emotional state is unstable, the alert's priority is increased.

[0240] Specific examples

[0241] 1. User statements and sentiment

[0242] User: "What's my cash card PIN?"

[0243] The device captures this audio and converts it into text.

[0244] The emotion engine analyzes the voice data and assesses that the user is feeling anxious.

[0245] 2. Server Analysis

[0246] The text and sentiment analysis results are sent to a server and analyzed by a generative AI model.

[0247] The model compares the data with past fraud cases and assesses the likelihood of fraud.

[0248] 3. Sending alerts

[0249] The server detects potential fraud and sends alerts to registered family members and the police.

[0250] Alert: "A potentially fraudulent conversation has been detected. The user is concerned. Please check immediately."

[0251] merit

[0252] This system enables real-time detection of fraudulent billing scams and rapid response. It can provide effective crime prevention measures, especially for individuals at high risk of becoming a victim of fraud, such as the elderly. Furthermore, by using an emotion engine, highly accurate fraud detection can be achieved by taking into account the user's mental state. This reduces false positives and improves the quality of responses.

[0253] The processing flow will be explained below.

[0254] Step 1:

[0255] Users install the app, launch it, and perform the initial setup, which includes registering contact information for family members, police, etc. This sets up where alerts will be sent if fraud is detected.

[0256] Step 2:

[0257] The device uses a microphone for voice capture to record the user's conversation in real time. The recorded voice data is processed within the device and converted into text data by a voice recognition system. For example, if a user says, "Tell me the PIN number for my cash card," this voice becomes the text data "Tell me the PIN number for my cash card."

[0258] Step 3:

[0259] The converted voice data is simultaneously analyzed by the device's emotion engine, which evaluates the user's emotional state from the voice data and determines whether the user is stressed, relaxed, or anxious. For example, if the user is assessed as being anxious, the level of anxiety is also recorded.

[0260] Step 4:

[0261] The converted text data and sentiment analysis results are sent from the device to a server. The server receives the text data and sentiment data and uses a generative AI model to analyze the text and assess the likelihood of fraud. The generative AI model compares the text with past fraud cases stored in a database to determine signs of fraud.

[0262] Step 5:

[0263] If the generative AI model's analysis determines that fraud is highly likely, the server generates an alert. This alert includes a statement that fraud is likely, as well as the user's comments and emotional state. For example, the alert could include the following: "A potentially fraudulent conversation has been detected. The user is feeling uneasy. Please check immediately."

[0264] Step 6:

[0265] The server generates an alert and sends it to pre-registered contacts (family members or the police). This alert allows the contacts to quickly identify possible fraud and take appropriate action. If the emotional state is unstable, the server sends an alert with a higher priority, encouraging a quicker response.

[0266] Step 7:

[0267] After an alert is sent, the server logs the results, including the fraudulent activity detected, the alert content, and the destination information, for later analysis and auditing.

[0268] Step 8:

[0269] Users who receive the alert can work with family members or law enforcement to take action to prevent the fraud from progressing, for example by reviewing the content of the call and taking specific steps to rule out the possibility of fraud.

[0270] Through this detailed process flow, users are protected from fraudulent billing scams with high accuracy and can respond quickly and effectively. The addition of an emotion engine enables advanced fraud detection that takes into account the user's mental state, improving overall safety and reliability.

[0271] Example 2

[0272] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0273] In today's world, elderly people and those unfamiliar with technology are increasingly being targeted by fraudulent billing scams. This puts many people at risk of financial loss. However, current security systems have difficulty detecting fraudulent activity in real time and taking appropriate action. Furthermore, because they do not take emotional states into account, they are prone to false positives and limited in the quality of the response. There is a need to solve these problems.

[0274] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0275] In this invention, the server includes: means for allowing a user to perform initial setup and register contact information; means for capturing the user's voice in real time and converting it into text; means for detecting the user's emotional state from the voice data and text data; means for transmitting the converted text data and emotion analysis results to the server; means for analyzing the text using a generative AI model and evaluating the possibility of fraud; and means for sending an alert to pre-registered contacts if the possibility of fraud is detected. This makes it possible to monitor user conversations in real time, detect fraud risks with high accuracy, and respond quickly.

[0276] "Means for users to set up and register contact information" refers to the process by which, after installing the application, users enter contact information for themselves and related parties and store it in a local database for subsequent data transmission and notifications.

[0277] "Means for capturing the user's voice in real time and converting it into text" refers to a function that uses the device's built-in microphone to record the user's speech and converts the voice data into a corresponding text format using voice recognition technology.

[0278] "Means for detecting a user's emotional state from voice data and text data" refers to technology that analyzes the content of a user's speech and voice characteristics to identify the emotions the user is feeling (e.g., anxiety, stress, relaxation).

[0279] "Means for transmitting the converted text data and emotion analysis results to the server" refers to a mechanism for transmitting the text data and emotion state acquired and analyzed on the device to the server using a secure communication protocol.

[0280] "Using a generative AI model to analyze the text and assess the likelihood of fraud" refers to the process of inputting received text data into an AI model based on a neural network or other advanced algorithm to assess the risk of fraud.

[0281] "Means of sending alerts to pre-registered contacts if possible fraudulent activity is detected" refers to a function that sends notifications to pre-registered emergency contacts (e.g., family members, police) if the evaluation results indicate a high possibility of fraudulent activity.

[0282] MODE FOR CARRYING OUT THE INVENTION

[0283] This invention is a system that constantly monitors users' conversations and emotions to detect fraudulent billing scams in real time, and sends alerts to family members and the police if there is a possibility of fraud. This system uses a generative AI model and an emotion engine to predict and detect fraud damage with high accuracy.

[0284] Installing and setting up the dedicated app

[0285] The user first installs the dedicated app and performs the initial setup, which includes registering the user's own contact information as well as emergency contact information for family members, police, etc. At this stage, the registered contact information is securely stored in the device's local database. The information entered by the user will be used for subsequent data transmission and alert generation.

[0286] Voice capture and text transcription

[0287] The device is equipped with a microphone for voice capture, which records the user's conversation in real time. This voice data is converted into text using voice recognition technology. A specific technology that can be used is Google Cloud Speech-to-Text. For example, if a user says, "Tell me the PIN number for my cash card," this voice is converted into text data: "Tell me the PIN number for my cash card."

[0288] emotion recognition

[0289] The converted voice data is analyzed by the device's emotion engine. A specific example of an emotion engine is IBM Watson's Natural Language Understanding, which can identify the user's emotional state (e.g., anxiety, stress, relaxation, etc.) from the content of the user's speech and voice characteristics. For example, the text and voice of a statement such as "Tell me my cash card PIN number" are analyzed and the user is evaluated as feeling anxious.

[0290] Sending data to the server

[0291] The sentiment analysis results and text data are sent to the server using a secure communication protocol such as HTTPS. The data sent includes the converted text data and the emotional state, allowing even sensitive information to be handled safely.

[0292] Potential fraud analysis

[0293] The server feeds the received data into a generative AI model (e.g., GPT-4) to assess the likelihood of fraud, for example using the following prompt:

[0294] "Analyze the following conversation and assess the likelihood of fraud. The user's emotion is 'anxiety'. Conversation: 'Please tell me your cash card PIN number.'"

[0295] The analysis results obtained in this way are compared with a database of past fraud cases to detect signs of fraud.

[0296] Generate and send alerts

[0297] If the generative AI model's analysis determines that fraud is likely, the server generates an alert. This alert includes information about the potential fraud, the content of the conversation, and the user's emotional state. The alert is sent to pre-registered contacts (e.g., family, police). The alert notification may include a message such as:

[0298] "A potentially fraudulent conversation has been detected. The user is concerned. Please check immediately."

[0299] This system monitors user conversations in real time, detects fraud risks with high accuracy, and responds quickly. It can provide effective crime prevention measures, especially for individuals at high risk of becoming a victim of fraud, such as the elderly. By using an emotion engine, highly accurate fraud detection is achieved that takes into account the user's mental state, reducing false positives and improving the quality of responses.

[0300] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0301] Step 1: Install and set up the dedicated app

[0302] The user installs the app. After installation, the app displays an initial setup screen where the user enters contact information for themselves, their family, the police, etc. The entered contact information is saved in the device's local database. The entered data is used in subsequent processes of the app.

[0303] Input: User's contact information

[0304] Output: Contact information stored in a local database

[0305] Specific behavior:

[0306] The user installs the app and accesses the initial setup screen.

[0307] You will be prompted to enter contact information for your family members.

[0308] When the user enters contact information and presses the "Save" button, the contact information is saved to a local database.

[0309] Step 2: Capture audio and convert it to text

[0310] The device uses a built-in microphone to record the user's speech in real time, and this voice data is converted to text using the Google Cloud Speech-to-Text API.

[0311] Input: User's voice data

[0312] Output: Text data

[0313] Specific behavior:

[0314] When the user starts speaking, the device's microphone starts capturing it.

[0315] For example, voice data saying "Tell me the PIN number for my cash card" is acquired.

[0316] The voice data is passed to the Google Cloud Speech-to-Text API and converted into text: "What is my cash card PIN?"

[0317] Step 3: Emotion Recognition

[0318] The device passes the converted text data to an emotion engine that uses IBM Watson's Natural Language Understanding to detect the user's emotional state from the voice and text data.

[0319] Input: Text data and audio data

[0320] Output: User's emotional state

[0321] Specific behavior:

[0322] The text data "Please tell me the PIN number for my cash card" is input to the emotion engine.

[0323] The emotion engine analyzes the tone and content of the user's voice.

[0324] As a result of the analysis, it is detected that the user is feeling "anxiety."

[0325] Step 4: Sending data to the server

[0326] The device sends the converted text data and the emotion analysis results to the server using the HTTP protocol.

[0327] Input: Text data and sentiment analysis results

[0328] Output: Data sent to the server

[0329] Specific behavior:

[0330] The text data generated on the device, "Please tell me your cash card PIN number" and the emotional state "anxiety," are structured in JSON format.

[0331] This data is sent to the server using the HTTPS protocol.

[0332] Step 5: Fraud Potential Analysis

[0333] The server inputs the received data into a generative AI model (e.g., GPT-4), which compares it with a database of past fraud cases to assess the likelihood of fraud.

[0334] Input: Text data, sentiment analysis results

[0335] Output: Fraud likelihood assessment result

[0336] Specific behavior:

[0337] The server generates a prompt. For example, "Analyze the following conversation and assess the likelihood of fraud. The user's emotion is detected as 'anxiety.' Conversation: 'Please tell me your cash card PIN number.'"

[0338] Text data along with a prompt sentence is input into the generative AI model.

[0339] The generative AI model performs the analysis and returns a result that indicates there is a high probability of fraud.

[0340] Step 6: Generate and send an alert

[0341] If the server determines that fraud is likely, it generates an alert. This alert includes information about the potential fraud, the content of the conversation, and the user's emotional state. The alert is sent to pre-registered contacts (family members or the police).

[0342] Input: Fraud possibility assessment results, registered contact information

[0343] Output: Alert notification sent

[0344] Specific behavior:

[0345] The server generates an alert message saying, "A potentially fraudulent conversation has been detected. The user is concerned. Please check immediately."

[0346] Alerts will be sent via email and SMS based on the registered contact information.

[0347] This system makes it possible to monitor user conversations in real time, detect fraud risks with high accuracy, and respond quickly.

[0348] (Application example 2)

[0349] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0350] Elderly people and those living alone are at increased risk of falling victim to fraudulent billing scams. These fraudulent activities are becoming more sophisticated, making simple fraud detection systems insufficient. Furthermore, more accurate judgments are required by taking into account the user's emotional state, but current systems are lacking in this area. Therefore, the object of this invention is to provide a security system that constantly monitors the user's conversations and emotions, detects fraudulent activities in real time with high accuracy, and responds quickly and appropriately.

[0351] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing a user's voice in real time and converting it into text, means for analyzing the text using a generative AI model and evaluating the possibility of fraud, means for sending an alert to pre-registered contacts when the possibility of fraud is detected, means for evaluating the user's emotional state using an emotion analysis engine, and means for including the user's emotional state in the alert along with the possibility of fraud. This enables highly accurate fraud detection and rapid response that takes the user's mental state into consideration.

[0352] "Means for capturing and converting user speech into text in real time" refers to a combination of devices and algorithms for quickly capturing speech emitted by a user and converting it into text information.

[0353] "Means for analyzing said text using a generative AI model to assess the likelihood of fraudulent activity" refers to a system that uses AI technology to analyze text data and determine whether it is related to fraudulent activity.

[0354] "A means of sending alerts to pre-registered contacts when possible fraudulent activity is detected" is a function that sends an alert to pre-set contacts such as family members or the police when signs of fraud are detected.

[0355] "Means for assessing a user's emotional state using an emotion analysis engine" refers to technology that analyzes voice and text data to recognize and evaluate the emotions felt by the user.

[0356] The "means for including the user's emotional state along with the possibility of fraud in the alert" is a mechanism for including the user's emotional state in a warning message indicating the possibility of fraud.

[0357] This invention is a system that monitors user conversations and emotions in real time, detects fraudulent billing with high accuracy, and promptly sends an alert. Specific embodiments for implementing this invention will be described below.

[0358] 1. System Configuration

[0359] Audio capture device:

[0360] Devices such as smartphones have built-in microphones that are used to capture the user's voice in real time.

[0361] Speech-to-text methods:

[0362] The captured audio data is converted to text using the Google Cloud Speech-to-Text API, which makes the audio information available as text.

[0363] Analysis methods using generative AI models:

[0364] The converted text data is then analyzed using a generative AI model, such as OpenAI's GPT-3, which compares it with a database of past fraud cases to assess the likelihood of fraud.

[0365] Sentiment Analysis Engine:

[0366] It uses IBM Watson's emotion analysis engine to analyze the user's emotional state, for example, assessing the user's stress, anxiety, or level of relaxation from voice data.

[0367] Alerts sent via:

[0368] If a fraudulent activity is assessed as likely, an alert will be sent to pre-registered contacts such as family members or the police, and the alert will include information indicating the possibility of fraud and the user's emotional state.

[0369] 2. Specific examples of processing

[0370] Capturing user conversations and emotions

[0371] Suppose a user says into their smartphone, "What is my PIN number for my cash card?" The smartphone's microphone captures this speech and sends it in real time to the Google Cloud Speech-to-Text API, where it is converted into text.

[0372] Emotion Analysis

[0373] The converted text and voice data is sent to IBM Watson's sentiment analysis engine to assess whether the user is feeling anxious.

[0374] Fraud Analysis

[0375] A generative AI model (OpenAI GPT-3) analyzes the converted text data and compares it with a database of past fraud cases. For example, the following prompt sentence is used:

[0376] Example prompt sentence:

[0377] Please rate the likelihood that the following conversation text is a scam.

[0378] Text: What is your cash card PIN number?

[0379] If the generative AI model's analysis determines that there is a high possibility of fraud, the system will generate an alert.

[0380] Sending alerts

[0381] The alert will be sent to pre-registered family members and the police. The alert content will include information about the possibility of fraud and the user's concerns. For example, an alert may be sent with the content, "A potentially fraudulent conversation has been detected. The user is concerned. Please check immediately."

[0382] The system of the present invention can protect elderly people and those living alone from sophisticated fraud. Furthermore, by taking into account the user's emotional state, it is expected to achieve more accurate fraud detection and reduce false positives.

[0383] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0384] Step 1:

[0385] The user speaks into the smartphone. The smartphone's microphone captures the voice in real time. The input is voice data, and the output is the captured voice. For example, the user might record a voice saying, "Tell me my cash card PIN."

[0386] Step 2:

[0387] The smartphone sends the captured voice data to the Google Cloud Speech-to-Text API, which converts it into text data in real time. The input is the captured voice data, and the output is the text data corresponding to that voice. Specifically, the voice saying "Tell me my cash card PIN" is converted into text "Tell me my cash card PIN."

[0388] Step 3:

[0389] The converted text data and voice data are sent to IBM Watson's emotion analysis engine. The emotion analysis engine analyzes this data and evaluates the user's emotional state. The input is text data and voice data, and the output is analyzed emotional state data. In concrete terms, the text "Tell me my cash card PIN" is evaluated as indicating that the user is feeling anxious.

[0390] Step 4:

[0391] A generative AI model (OpenAI GPT-3) analyzes the converted text data. The input is the text data and the sentiment analysis results, and the output is an assessment of the likelihood of fraud. The prompt used is "Please assess the likelihood that the following conversational text is fraudulent. Text: Tell me your cash card PIN." The analysis results are obtained.

[0392] Step 5:

[0393] If the server determines that there is a high possibility of fraud, it generates an alert. The input is the result of the fraud possibility assessment and the user's emotional state data, and the output is the alert content. The alert includes information about the possibility of fraud and the user's anxiety. Specifically, the alert content is "A potentially fraudulent conversation has been detected. The user is feeling anxious. Please check immediately."

[0394] Step 6:

[0395] The server sends the alert content to pre-registered contacts such as family members and the police. The input is the alert content, and the output is the sent alert. Specifically, alerts are sent via email or messaging apps.

[0396] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0397] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0398] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0399] [Second embodiment]

[0400] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0401] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0402] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0403] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0404] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0405] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0406] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0407] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0408] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0409] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0410] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0411] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0412] overview

[0413] This invention is a system that constantly monitors users' conversations to detect fraudulent billing in real time and sends alerts to family members and the police if there is a possibility of fraud. This system includes an analysis function using a generative AI model, and can make highly accurate judgments by comparing the results with a database of past fraud cases.

[0414] Configuring Users

[0415] Users install a dedicated app and perform the initial setup, registering contact information such as family members and the police. This allows the device to be set up so that alerts are sent to the registered contacts if fraud is detected.

[0416] Audio Capture and Conversion

[0417] The device is equipped with a microphone for voice capture, which records the user's conversation in real time. This voice data is converted into text using voice recognition technology. For example, if a user says, "Tell me the PIN number for my cash card," the voice data is instantly converted into text data.

[0418] Analysis and Evaluation

[0419] The converted text is sent from the device to a server, which uses a generative AI model to analyze the text and assess its likelihood of fraud. The generative AI model then compares the text with a database of past fraud cases to determine whether it indicates fraud.

[0420] Sending alerts

[0421] If a potential fraudulent activity is detected, the server will send an alert to registered contacts. The alert will include the content of the potentially fraudulent conversation and a message urging immediate action. For example, a notification might say, "A potentially fraudulent conversation has been detected. Please check immediately."

[0422] Specific examples

[0423] 1. User comments

[0424] User: "What's my cash card PIN?"

[0425] The device captures this audio and converts it into text.

[0426] Converted text: "What is your cash card PIN?"

[0427] 2. Server Analysis

[0428] The text is sent to a server and analyzed by a generative AI model.

[0429] The model compares the data with past fraud cases and rates the likelihood of fraud highly.

[0430] 3. Sending alerts

[0431] The server detects potential fraud and sends alerts to registered family members and the police.

[0432] Alert: "A potentially fraudulent conversation has been detected. Please investigate immediately."

[0433] merit

[0434] This system enables real-time detection of fraudulent billing scams and rapid response, providing effective crime prevention measures, particularly for individuals at high risk of fraud, such as the elderly. Furthermore, the use of generative AI models can improve the accuracy of fraud detection and reduce false positives.

[0435] The processing flow will be explained below.

[0436] Step 1:

[0437] Users install the app, launch it, and perform the initial setup, which includes registering contact information for family members, police, etc. This sets up where alerts will be sent if fraud is detected.

[0438] Step 2:

[0439] The terminal uses a microphone for voice capture to record the user's conversation in real time. This voice data is immediately sent to a voice recognition system by a processing unit in the terminal and converted into text format. For example, if a user says, "Tell me the PIN number for my cash card," this voice is converted into text data, "Tell me the PIN number for my cash card."

[0440] Step 3:

[0441] The converted text data is sent from the device to a server, which receives the text data and begins analyzing it using a generative AI model.

[0442] Step 4:

[0443] The server analyzes the received text data using a generative AI model to assess the likelihood of fraudulent activity, which then matches the text with past fraud cases stored in a database and runs algorithms to detect signs of fraud.

[0444] Step 5:

[0445] If the generative AI model's analysis determines that fraud is likely, the server generates an alert, which includes a description of the potential fraud and the conversation.

[0446] Step 6:

[0447] The server generates an alert and sends it to pre-registered contacts, including the user's family and the police, with a message urging them to immediately investigate the fraudulent activity, such as "A potentially fraudulent conversation has been detected. Please investigate immediately."

[0448] Step 7:

[0449] After an alert is sent, the server logs the results, including the fraudulent activity detected, the alert content, and the destination information, for later analysis and auditing.

[0450] Step 8:

[0451] Users who receive the alert can work with their family or the police to take action to prevent the fraud from progressing, for example by taking specific measures such as verifying personal information with callers before providing it.

[0452] This process flow protects users from fraudulent billing and allows for a quick response.

[0453] Example 1

[0454] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0455] Fraudulent activities, such as fictitious billing scams, have a serious impact, especially on vulnerable individuals such as the elderly. To address this issue, a system that can detect fraud in real time and respond quickly is needed. However, conventional methods have low accuracy in detecting fraud, and are prone to false positives and delays. Therefore, a method is needed to detect fraud in real time with high accuracy and respond quickly.

[0456] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0457] In this invention, the server includes a means for capturing a user's voice in real time and converting it into text, a means for analyzing the text using a generative AI model to evaluate the likelihood of fraud, a means for sending an alert to pre-registered contacts if a likelihood of fraud is detected, a means for comparing the text with a database of past fraud cases, and a means for sending the text to the server using encrypted communication, thereby enabling fraud to be detected with high accuracy in real time and alerting family members and the police quickly.

[0458] "Means for capturing and converting a user's voice into text in real time" refers to a device or software that instantly records the voice data spoken by a user and converts it into text data using voice recognition technology.

[0459] "Using a generative AI model to analyze said text and assess the likelihood of fraud" refers to technology that uses a machine learning model to analyze text data and determine the likelihood of fraud based on specific patterns or characteristics of fraud.

[0460] "Means for sending alerts to pre-defined contacts when potential fraud is detected" refers to a method or device for sending a warning message to pre-defined family or law enforcement contacts when signs of fraud are detected.

[0461] "Means for comparing the text with a database of past fraud cases" refers to technology for determining fraudulent activity by referring to a database that records past fraud cases and comparing and matching the text data to be analyzed.

[0462] "Means for transmitting the text to the server using encrypted communication" refers to a method that uses data encryption technology to securely transmit text data to the server.

[0463] overview

[0464] This invention is a system that detects fraudulent billing in real time by constantly monitoring the user's voice and sends an alert to family members and the police if there is a possibility of fraud. This system performs analysis using a generative AI model and compares the results with a database of past fraud cases, enabling highly accurate judgments.

[0465] System Configuration

[0466] This system is realized mainly using the following hardware and software.

[0467] Hardware

[0468] Device: A smartphone or tablet with a microphone for audio capture.

[0469] Server: A high-performance computer for running generative AI models.

[0470] software

[0471] Generative AI model: A machine learning model that analyzes fraudulent activity.

[0472] Speech recognition software: For example, using the Google Speech-to-Text API.

[0473] Database: A database containing past fraud cases.

[0474] Specific processing of the program

[0475] User Preferences

[0476] Users install a dedicated app on their device and register contact information for family members and the police, so that if fraud is detected, an alert is automatically sent to the designated contacts.

[0477] Audio capture and transcription

[0478] The device's built-in microphone constantly records the user's speech and converts this voice data into text data in real time. For example, if a user says, "Tell me my cash card PIN," the speech is converted into text using voice recognition software.

[0479] Sending and analyzing text data

[0480] The converted text data is sent to a server using encrypted communications. The server then analyzes the received text data using a generative AI model to assess the likelihood of fraud. This analysis includes a process of comparing it with a database of past fraud cases.

[0481] Sending alerts

[0482] If a fraudulent activity is detected, the server will send an alert to registered family members or the police. The alert will include the details of the potentially fraudulent conversation and a message urging immediate action. For example, a notification such as "A potentially fraudulent conversation has been detected. Please check immediately."

[0483] Examples and prompts

[0484] Specific examples

[0485] 1. User utterance: The user says, "Tell me the PIN number for my cash card."

[0486] What the device does: It captures this audio and converts it to text using speech recognition software.

[0487] Converted text: "What is your cash card PIN?"

[0488] 2. Server analysis: The converted text is sent to the server and analyzed using the generative AI model.

[0489] Analysis results: The generative AI model compares the data with a database of past fraud cases and evaluates the likelihood of fraudulent activity.

[0490] 3. Send alert: The server determines that there is a high possibility of fraud and sends an alert to pre-registered family members and the police.

[0491] Alert: "A potentially fraudulent conversation has been detected. Please investigate immediately."

[0492] Prompt Sentence Examples

[0493] To perform analysis, enter the following prompt into the generative AI model:

[0494] "Judge whether the following text is likely to be fraudulent: 'Tell me your bank card PIN'"

[0495] As described above, the present invention is a system that aims to analyze user conversations in real time, detect fraudulent acts with high accuracy, and respond quickly.

[0496] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0497] Step 1: Initial User Setup

[0498] Users install a dedicated app on their device. When they launch the app for the first time, they register contact information for family members and the police, which allows the app to send prompt notifications when fraud is detected. The input is the contact information entered by the user, and the output is that it is saved on the server.

[0499] Input: Family and police contact information

[0500] Output: Contact information is saved on the server

[0501] Specific operation: The user opens the app, enters the required information on the contact registration screen, and presses the save button.

[0502] Step 2: Capture audio

[0503] The device uses a built-in microphone to capture the user's speech in real time. The captured voice data is temporarily stored on the device. The input is the user's voice and the output is the captured voice data.

[0504] Input: User spoken words

[0505] Output: Captured audio data

[0506] What it does: The device runs the app in the background, and the microphone constantly captures audio and stores it in temporary storage.

[0507] Step 3: Convert audio data to text

[0508] The device converts the captured voice data into text data in real time using voice recognition software (e.g., Google Speech-to-Text API). The input is the captured voice data, and the output is text data.

[0509] Input: Captured audio data

[0510] Output: Converted text data

[0511] Specific operation: The saved voice data is converted into text data by voice recognition software.

[0512] Step 4: Sending text data

[0513] The terminal sends the converted text data to the server using encrypted communication. The input is the converted text data, and the output is the data sent to the server.

[0514] Input: Converted text data

[0515] Output: Text data is sent to the server

[0516] Specific operation: Once text data is generated, it is automatically sent to the server, and SSL / TLS encrypted communication is used.

[0517] Step 5: Analyzing the text data

[0518] The server analyzes the received text data using a generative AI model to assess the likelihood of fraud. The generative AI model makes its judgment based on previously learned fraud patterns. The input is the text data sent to the server, and the output is the assessment result of fraud.

[0519] Input: Text data sent to the server

[0520] Output: Fraud assessment results

[0521] Specific operation: The server inputs the received text data into a generative AI model, which then analyzes the data by comparing it with a database of past fraud cases.

[0522] Step 6: Generate fraud findings

[0523] The server receives the analysis results from the generative AI model and evaluates whether the likelihood of fraud is high or low. If it is judged to be high, it is considered fraud. The input is the analyzed text data, and the output is the fraud judgment result.

[0524] Input: Parsed text data

[0525] Output: Fraud detection result

[0526] Specific operation: The server receives the analysis evaluation results and identifies the possibility of fraud using a number or tag.

[0527] Step 7: Sending an alert

[0528] If the server determines that there is a high possibility of fraud, it will send an alert to registered family members or the police. The input is the fraud judgment result and pre-registered contact information, and the output is an alert message.

[0529] Input: Fraud detection result, registered contact information

[0530] Output: Alert message

[0531] What happens: The server generates an alert message and sends it via SMS, email, or in-app notification. Notification content: "A potentially fraudulent conversation has been detected. Please check immediately."

[0532] (Application example 1)

[0533] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0534] In recent years, there has been a sharp increase in fraudulent activities such as fictitious billing scams and bank transfer scams, with elderly people and those in socially vulnerable positions being particularly targeted. Conventional fraud prevention measures are limited to crime prevention education and warnings, and there are limitations to detecting fraud in real time and responding quickly, making it difficult to quickly prevent fraud damage, so a solution is needed.

[0535] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0536] In this invention, the server includes means for capturing a user's voice in real time and converting it into text, means for analyzing the text using a generative AI model to evaluate the possibility of fraud, means for sending an alert to pre-registered contacts when a possibility of fraud is detected, means for being installed on the smartphone and monitoring the user's calls in real time, means for recording the user's conversations and analyzing the content using a generative AI model, means for comparing the content with a database of past fraud cases to evaluate signs of fraud, and communication means for instantly sending the alert, thereby enabling fraud detection in real time and rapid response.

[0537] "Real-time voice capture" refers to technology that instantly records the voice spoken by the user and acquires the voice data.

[0538] "Convert to text" refers to the process of converting audio data into written information.

[0539] A "generative AI model" is an artificial intelligence algorithm or machine learning model that has been trained for a specific task.

[0540] "Analysis" is the process of evaluating and understanding data to understand its meaning and content.

[0541] "Fraudulent Potential" refers to an indicator or assessment of the likelihood that an action or statement may be fraudulent.

[0542] "Alert" means a notice or warning of an emergency or caution.

[0543] "Pre-registered contacts" refers to the phone numbers and email addresses that the user has set up in advance to receive notifications.

[0544] A "smartphone" refers to a mobile information terminal that has communication capabilities, internet connectivity, and can run applications.

[0545] "Real-time call monitoring" refers to the process of instantly monitoring and tracking user calls and communications.

[0546] "Recording audio" refers to the act of saving a user's speech or conversation on a recording medium.

[0547] "Content analysis" refers to the process of examining recorded data to understand its meaning and intent.

[0548] A "database of past fraud cases" refers to a collection of information that has accumulated data on fraud cases that have occurred in the past.

[0549] "Assessing for fraudulent indicators" refers to the process of determining the likelihood that a particular action or statement may be fraudulent and drawing a consequence.

[0550] "Communication means" refers to the technology and infrastructure for transmitting information, including the Internet and mobile communications.

[0551] System configuration

[0552] The present invention is implemented as a security application installed on a smartphone, which monitors users' calls in real time and performs a series of processes to detect fraudulent activity.

[0553] Hardware and software used

[0554] Smartphone: microphone, internet connection

[0555] Speech recognition software: Google Cloud Speech-to-Text API

[0556] Generative AI model: OpenAI GPT-4

[0557] Database: Database of past fraud cases

[0558] Communication Method: Uses Twilio API to send alerts via SMS and phone

[0559] Data processing and calculation

[0560] Audio Capture

[0561] The user's call is recorded in real time through the smartphone's microphone, and the audio data is captured in high quality and is used directly for the next step, which is to convert it into text.

[0562] Speech-to-text

[0563] The recorded voice data is converted into text data in real time using the Google Cloud Speech-to-Text API, so the words spoken by the user are recorded as text and can be used directly in subsequent analysis processes.

[0564] Text analytics

[0565] The generative AI model (OpenAI GPT-4) uses text data to analyze the content of utterances, using a pre-filled prompt, "Is this likely to be fraud?", to accurately assess the likelihood of fraud.

[0566] Comparison with a database of past fraud cases

[0567] The analyzed text data is compared with a database of past fraud cases, which contains a large number of past fraud methods and cases, and based on this, the system evaluates signs of fraud with high accuracy.

[0568] Sending alerts

[0569] If a fraudulent activity is deemed likely, an alert will be sent immediately to the user's pre-registered contacts (family, police, etc.) using the Twilio API. The alert will include the contents of the detected conversation and a message urging immediate action.

[0570] Specific examples

[0571] 1. Audio Capture

[0572] A user says on the phone, "Tell me the PIN number for my cash card."

[0573] The smartphone's microphone records this audio.

[0574] 2. Text Conversion

[0575] The recorded voice is converted into "Please tell me my cash card PIN number" via the Google Cloud Speech-to-Text API.

[0576] 3. Text Analysis

[0577] A generative AI model (OpenAI GPT-4) analyzes the text and evaluates it against the prompt, "Is this a possible scam?"

[0578] Evaluation result: "High probability of fraud"

[0579] 4. Sending alerts

[0580] Alerts are instantly sent to the user's family and police using the Twilio API.

[0581] Alert: "A potentially fraudulent conversation has been detected. Please investigate immediately."

[0582] Prompt Sentence Examples

[0583] text

[0584] Fraud prevention system:

[0585] Text generated from user's call data: "What is my cash card PIN?"

[0586] Prompt: "Could this be a scam?"

[0587] In this way, the present invention is able to detect fraudulent activity in real time and respond quickly.

[0588] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0589] Step 1:

[0590] A user initiates a call through a smartphone. The user's speech is captured in real time by the smartphone's microphone and collected as audio data. The input is the user's speech, and the output is the recorded audio data.

[0591] Step 2:

[0592] The device sends the captured audio data to the Google Cloud Speech-to-Text API, where it is analyzed and converted to text. The input is the original audio data, and the output is the converted text data.

[0593] Step 3:

[0594] The device sends the converted text data to a generative AI model (OpenAI GPT-4), which analyzes the text data using a pre-defined prompt: "Is this likely to be fraud?" The input is the text data and the prompt, and the output is an assessment of the likelihood of fraud.

[0595] Step 4:

[0596] The server receives the evaluation results from the generative AI model and compares them with a database of past fraud cases, where patterns and features that match fraudulent activity are further evaluated. The input is the text analysis results and database information, and the output is the final evaluation result.

[0597] Step 5:

[0598] If fraud is determined to be likely, the server uses the Twilio API to send an alert to pre-registered contacts, where the input is the evaluation result and registered contact information, and the output is the sent alert message.

[0599] Step 6:

[0600] Finally, the user's family or the police receive the alert and quickly confirm and take necessary action. In this step, the input is the alert message and the output is the confirmation of the message and the corresponding action.

[0601] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0602] overview

[0603] This invention is a system that constantly monitors users' conversations and emotions to detect fraudulent billing scams in real time, and sends alerts to family members and the police if there is a possibility of fraud. This system uses a generative AI model and an emotion engine to predict and detect fraud damage with high accuracy.

[0604] Configuring Users

[0605] Users install the app and perform the initial setup, which includes registering contact information for family members, police, etc. This sets up where alerts will be sent if fraud is detected.

[0606] Audio Capture and Conversion

[0607] The device is equipped with a microphone for voice capture, which records the user's conversation in real time. This voice data is converted into text using voice recognition technology. For example, if a user says, "Tell me the PIN number for my cash card," this voice is converted into text data: "Tell me the PIN number for my cash card."

[0608] emotion recognition

[0609] The converted voice data is analyzed by the device's emotion engine, which detects the user's emotional state from the voice data. This analysis evaluates whether the user is feeling stressed, relaxed, anxious, etc.

[0610] Analysis and Evaluation

[0611] The converted text data and the results of the sentiment analysis are sent from the device to a server. The server uses a generative AI model to analyze the text data and evaluate the likelihood of fraud. The generative AI model compares the text with past fraud cases stored in a database to detect signs of fraud. The user's emotional state also influences the evaluation.

[0612] Sending alerts

[0613] If the generative AI model's analysis determines that fraud is highly likely, the server generates an alert. This alert includes information about the potential fraud, the content of the conversation, and the user's emotional state. The alert is sent to pre-registered contacts (family members or police). If the user's emotional state is unstable, the alert's priority is increased.

[0614] Specific examples

[0615] 1. User statements and sentiment

[0616] User: "What's my cash card PIN?"

[0617] The device captures this audio and converts it into text.

[0618] The emotion engine analyzes the voice data and assesses that the user is feeling anxious.

[0619] 2. Server Analysis

[0620] The text and sentiment analysis results are sent to a server and analyzed by a generative AI model.

[0621] The model compares the data with past fraud cases and assesses the likelihood of fraud.

[0622] 3. Sending alerts

[0623] The server detects potential fraud and sends alerts to registered family members and the police.

[0624] Alert: "A potentially fraudulent conversation has been detected. The user is concerned. Please check immediately."

[0625] merit

[0626] This system enables real-time detection of fraudulent billing scams and rapid response. It can provide effective crime prevention measures, especially for individuals at high risk of becoming a victim of fraud, such as the elderly. Furthermore, by using an emotion engine, highly accurate fraud detection can be achieved by taking into account the user's mental state. This reduces false positives and improves the quality of responses.

[0627] The processing flow will be explained below.

[0628] Step 1:

[0629] Users install the app, launch it, and perform the initial setup, which includes registering contact information for family members, police, etc. This sets up where alerts will be sent if fraud is detected.

[0630] Step 2:

[0631] The device uses a microphone for voice capture to record the user's conversation in real time. The recorded voice data is processed within the device and converted into text data by a voice recognition system. For example, if a user says, "Tell me the PIN number for my cash card," this voice becomes the text data "Tell me the PIN number for my cash card."

[0632] Step 3:

[0633] The converted voice data is simultaneously analyzed by the device's emotion engine, which evaluates the user's emotional state from the voice data and determines whether the user is stressed, relaxed, or anxious. For example, if the user is assessed as being anxious, the level of anxiety is also recorded.

[0634] Step 4:

[0635] The converted text data and sentiment analysis results are sent from the device to a server. The server receives the text data and sentiment data and uses a generative AI model to analyze the text and assess the likelihood of fraud. The generative AI model compares the text with past fraud cases stored in a database to determine signs of fraud.

[0636] Step 5:

[0637] If the generative AI model's analysis determines that fraud is highly likely, the server generates an alert. This alert includes a statement that fraud is likely, as well as the user's comments and emotional state. For example, the alert could include the following: "A potentially fraudulent conversation has been detected. The user is feeling uneasy. Please check immediately."

[0638] Step 6:

[0639] The server generates an alert and sends it to pre-registered contacts (family members or the police). This alert allows the contacts to quickly identify possible fraud and take appropriate action. If the emotional state is unstable, the server sends an alert with a higher priority, encouraging a quicker response.

[0640] Step 7:

[0641] After an alert is sent, the server logs the results, including the fraudulent activity detected, the alert content, and the destination information, for later analysis and auditing.

[0642] Step 8:

[0643] Users who receive the alert can work with family members or law enforcement to take action to prevent the fraud from progressing, for example by reviewing the content of the call and taking specific steps to rule out the possibility of fraud.

[0644] Through this detailed process flow, users are protected from fraudulent billing scams with high accuracy and can respond quickly and effectively. The addition of an emotion engine enables advanced fraud detection that takes into account the user's mental state, improving overall safety and reliability.

[0645] Example 2

[0646] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0647] In today's world, elderly people and those unfamiliar with technology are increasingly being targeted by fraudulent billing scams. This puts many people at risk of financial loss. However, current security systems have difficulty detecting fraudulent activity in real time and taking appropriate action. Furthermore, because they do not take emotional states into account, they are prone to false positives and limited in the quality of the response. There is a need to solve these problems.

[0648] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0649] In this invention, the server includes: means for allowing a user to perform initial setup and register contact information; means for capturing the user's voice in real time and converting it into text; means for detecting the user's emotional state from the voice data and text data; means for transmitting the converted text data and emotion analysis results to the server; means for analyzing the text using a generative AI model and evaluating the possibility of fraud; and means for sending an alert to pre-registered contacts if the possibility of fraud is detected. This makes it possible to monitor user conversations in real time, detect fraud risks with high accuracy, and respond quickly.

[0650] "Means for users to set up and register contact information" refers to the process by which, after installing the application, users enter contact information for themselves and related parties and store it in a local database for subsequent data transmission and notifications.

[0651] "Means for capturing the user's voice in real time and converting it into text" refers to a function that uses the device's built-in microphone to record the user's speech and converts the voice data into a corresponding text format using voice recognition technology.

[0652] "Means for detecting a user's emotional state from voice data and text data" refers to technology that analyzes the content of a user's speech and voice characteristics to identify the emotions the user is feeling (e.g., anxiety, stress, relaxation).

[0653] "Means for transmitting the converted text data and emotion analysis results to the server" refers to a mechanism for transmitting the text data and emotion state acquired and analyzed on the device to the server using a secure communication protocol.

[0654] "Using a generative AI model to analyze the text and assess the likelihood of fraud" refers to the process of inputting received text data into an AI model based on a neural network or other advanced algorithm to assess the risk of fraud.

[0655] "Means of sending alerts to pre-registered contacts if possible fraudulent activity is detected" refers to a function that sends notifications to pre-registered emergency contacts (e.g., family members, police) if the evaluation results indicate a high possibility of fraudulent activity.

[0656] MODE FOR CARRYING OUT THE INVENTION

[0657] This invention is a system that constantly monitors users' conversations and emotions to detect fraudulent billing scams in real time, and sends alerts to family members and the police if there is a possibility of fraud. This system uses a generative AI model and an emotion engine to predict and detect fraud damage with high accuracy.

[0658] Installing and setting up the dedicated app

[0659] The user first installs the dedicated app and performs the initial setup, which includes registering the user's own contact information as well as emergency contact information for family members, police, etc. At this stage, the registered contact information is securely stored in the device's local database. The information entered by the user will be used for subsequent data transmission and alert generation.

[0660] Voice capture and text transcription

[0661] The device is equipped with a microphone for voice capture, which records the user's conversation in real time. This voice data is converted into text using voice recognition technology. A specific technology that can be used is Google Cloud Speech-to-Text. For example, if a user says, "Tell me the PIN number for my cash card," this voice is converted into text data: "Tell me the PIN number for my cash card."

[0662] emotion recognition

[0663] The converted voice data is analyzed by the device's emotion engine. A specific example of an emotion engine is IBM Watson's Natural Language Understanding, which can identify the user's emotional state (e.g., anxiety, stress, relaxation, etc.) from the content of the user's speech and voice characteristics. For example, the text and voice of a statement such as "Tell me my cash card PIN number" are analyzed and the user is evaluated as feeling anxious.

[0664] Sending data to the server

[0665] The sentiment analysis results and text data are sent to the server using a secure communication protocol such as HTTPS. The data sent includes the converted text data and the emotional state, allowing even sensitive information to be handled safely.

[0666] Potential fraud analysis

[0667] The server feeds the received data into a generative AI model (e.g., GPT-4) to assess the likelihood of fraud, for example using the following prompt:

[0668] "Analyze the following conversation and assess the likelihood of fraud. The user's emotion is 'anxiety'. Conversation: 'Please tell me your cash card PIN number.'"

[0669] The analysis results obtained in this way are compared with a database of past fraud cases to detect signs of fraud.

[0670] Generate and send alerts

[0671] If the generative AI model's analysis determines that fraud is likely, the server generates an alert. This alert includes information about the potential fraud, the content of the conversation, and the user's emotional state. The alert is sent to pre-registered contacts (e.g., family, police). The alert notification may include a message such as:

[0672] "A potentially fraudulent conversation has been detected. The user is concerned. Please check immediately."

[0673] This system monitors user conversations in real time, detects fraud risks with high accuracy, and responds quickly. It can provide effective crime prevention measures, especially for individuals at high risk of becoming a victim of fraud, such as the elderly. By using an emotion engine, highly accurate fraud detection is achieved that takes into account the user's mental state, reducing false positives and improving the quality of responses.

[0674] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0675] Step 1: Install and set up the dedicated app

[0676] The user installs the app. After installation, the app displays an initial setup screen where the user enters contact information for themselves, their family, the police, etc. The entered contact information is saved in the device's local database. The entered data is used in subsequent processing of the app.

[0677] Input: User's contact information

[0678] Output: Contact information stored in a local database

[0679] Specific behavior:

[0680] The user installs the app and accesses the initial setup screen.

[0681] You will be prompted to enter contact information for your family members.

[0682] When the user enters contact information and presses the "Save" button, the contact information is saved to a local database.

[0683] Step 2: Capture audio and convert it to text

[0684] The device uses a built-in microphone to record the user's speech in real time, and this voice data is converted to text using the Google Cloud Speech-to-Text API.

[0685] Input: User's voice data

[0686] Output: Text data

[0687] Specific behavior:

[0688] When the user starts speaking, the device's microphone starts capturing it.

[0689] For example, voice data saying "Tell me the PIN number for my cash card" is acquired.

[0690] The voice data is passed to the Google Cloud Speech-to-Text API and converted into text: "What is my cash card PIN?"

[0691] Step 3: Emotion Recognition

[0692] The device passes the converted text data to an emotion engine that uses IBM Watson's Natural Language Understanding to detect the user's emotional state from the voice and text data.

[0693] Input: Text data and audio data

[0694] Output: User's emotional state

[0695] Specific behavior:

[0696] The text data "Please tell me the PIN number for my cash card" is input to the emotion engine.

[0697] The emotion engine analyzes the tone and content of the user's voice.

[0698] As a result of the analysis, it is detected that the user is feeling "anxiety."

[0699] Step 4: Sending data to the server

[0700] The device sends the converted text data and the emotion analysis results to the server using the HTTP protocol.

[0701] Input: Text data and sentiment analysis results

[0702] Output: Data sent to the server

[0703] Specific behavior:

[0704] The text data generated on the device, "Please tell me your cash card PIN number" and the emotional state "anxiety," are structured in JSON format.

[0705] This data is sent to the server using the HTTPS protocol.

[0706] Step 5: Fraud Potential Analysis

[0707] The server inputs the received data into a generative AI model (e.g., GPT-4), which compares it with a database of past fraud cases to assess the likelihood of fraud.

[0708] Input: Text data, sentiment analysis results

[0709] Output: Fraud likelihood assessment result

[0710] Specific behavior:

[0711] The server generates a prompt. For example, "Analyze the following conversation and assess the likelihood of fraud. The user's emotion is detected as 'anxiety.' Conversation: 'Please tell me your cash card PIN number.'"

[0712] Text data along with a prompt sentence is input into the generative AI model.

[0713] The generative AI model performs the analysis and returns a result that indicates there is a high probability of fraud.

[0714] Step 6: Generate and send an alert

[0715] If the server determines that fraud is likely, it generates an alert. This alert includes information about the potential fraud, the content of the conversation, and the user's emotional state. The alert is sent to pre-registered contacts (family members or the police).

[0716] Input: Fraud possibility assessment results, registered contact information

[0717] Output: Alert notification sent

[0718] Specific behavior:

[0719] The server generates an alert message saying, "A potentially fraudulent conversation has been detected. The user is concerned. Please check immediately."

[0720] Alerts will be sent via email and SMS based on the registered contact information.

[0721] This system makes it possible to monitor user conversations in real time, detect fraud risks with high accuracy, and respond quickly.

[0722] (Application example 2)

[0723] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0724] Elderly people and those living alone are at increased risk of falling victim to fraudulent billing scams. These fraudulent activities are becoming more sophisticated, making simple fraud detection systems insufficient. Furthermore, more accurate judgments are required by taking into account the user's emotional state, but current systems are lacking in this area. Therefore, the object of this invention is to provide a security system that constantly monitors the user's conversations and emotions, detects fraudulent activities in real time with high accuracy, and responds quickly and appropriately.

[0725] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing a user's voice in real time and converting it into text, means for analyzing the text using a generative AI model and evaluating the possibility of fraud, means for sending an alert to pre-registered contacts when the possibility of fraud is detected, means for evaluating the user's emotional state using an emotion analysis engine, and means for including the user's emotional state in the alert along with the possibility of fraud. This enables highly accurate fraud detection and rapid response that takes the user's mental state into consideration.

[0726] "Means for capturing and converting user speech into text in real time" refers to a combination of devices and algorithms for quickly capturing speech emitted by a user and converting it into text information.

[0727] "Means for analyzing said text using a generative AI model to assess the likelihood of fraudulent activity" refers to a system that uses AI technology to analyze text data and determine whether it is related to fraudulent activity.

[0728] "A means of sending alerts to pre-registered contacts when possible fraudulent activity is detected" is a function that sends an alert to pre-set contacts such as family members or the police when signs of fraud are detected.

[0729] "Means for assessing a user's emotional state using an emotion analysis engine" refers to technology that analyzes voice and text data to recognize and evaluate the emotions felt by the user.

[0730] The "means for including the user's emotional state along with the possibility of fraud in the alert" is a mechanism for including the user's emotional state in a warning message indicating the possibility of fraud.

[0731] This invention is a system that monitors user conversations and emotions in real time, detects fraudulent billing with high accuracy, and promptly sends an alert. Specific embodiments for implementing this invention will be described below.

[0732] 1. System Configuration

[0733] Audio capture device:

[0734] Devices such as smartphones have built-in microphones that are used to capture the user's voice in real time.

[0735] Speech-to-text methods:

[0736] The captured audio data is converted to text using the Google Cloud Speech-to-Text API, which makes the audio information available as text.

[0737] Analysis methods using generative AI models:

[0738] The converted text data is then analyzed using a generative AI model, such as OpenAI's GPT-3, which compares it with a database of past fraud cases to assess the likelihood of fraud.

[0739] Sentiment Analysis Engine:

[0740] It uses IBM Watson's emotion analysis engine to analyze the user's emotional state, for example, assessing the user's stress, anxiety, or level of relaxation from voice data.

[0741] Alerts sent via:

[0742] If a fraudulent activity is assessed as likely, an alert will be sent to pre-registered contacts such as family members or the police, and the alert will include information indicating the possibility of fraud and the user's emotional state.

[0743] 2. Specific examples of processing

[0744] Capturing user conversations and emotions

[0745] Suppose a user says into their smartphone, "What is my PIN number for my cash card?" The smartphone's microphone captures this speech and sends it in real time to the Google Cloud Speech-to-Text API, where it is converted into text.

[0746] Emotion Analysis

[0747] The converted text and voice data is sent to IBM Watson's sentiment analysis engine to assess whether the user is feeling anxious.

[0748] Fraud Analysis

[0749] A generative AI model (OpenAI GPT-3) analyzes the converted text data and compares it with a database of past fraud cases. For example, the following prompt sentence is used:

[0750] Example prompt sentence:

[0751] Please rate the likelihood that the following conversation text is a scam.

[0752] Text: What is your cash card PIN number?

[0753] If the generative AI model's analysis determines that there is a high possibility of fraud, the system will generate an alert.

[0754] Sending alerts

[0755] The alert will be sent to pre-registered family members and the police. The alert content will include information about the possibility of fraud and the user's concerns. For example, an alert may be sent with the content, "A potentially fraudulent conversation has been detected. The user is concerned. Please check immediately."

[0756] The system of the present invention can protect elderly people and those living alone from sophisticated fraud. Furthermore, by taking into account the user's emotional state, it is expected to achieve more accurate fraud detection and reduce false positives.

[0757] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0758] Step 1:

[0759] The user speaks into the smartphone. The smartphone's microphone captures the voice in real time. The input is voice data, and the output is the captured voice. For example, the user might record a voice saying, "Tell me my cash card PIN."

[0760] Step 2:

[0761] The smartphone sends the captured voice data to the Google Cloud Speech-to-Text API, which converts it into text data in real time. The input is the captured voice data, and the output is the text data corresponding to that voice. Specifically, the voice saying "Tell me my cash card PIN" is converted into text "Tell me my cash card PIN."

[0762] Step 3:

[0763] The converted text data and voice data are sent to IBM Watson's emotion analysis engine. The emotion analysis engine analyzes this data and evaluates the user's emotional state. The input is text data and voice data, and the output is analyzed emotional state data. In concrete terms, the text "Tell me my cash card PIN" is evaluated as indicating that the user is feeling anxious.

[0764] Step 4:

[0765] A generative AI model (OpenAI GPT-3) analyzes the converted text data. The input is the text data and the sentiment analysis results, and the output is an assessment of the likelihood of fraud. The prompt used is "Please assess the likelihood that the following conversational text is fraudulent. Text: Tell me your cash card PIN." The analysis results are obtained.

[0766] Step 5:

[0767] If the server determines that there is a high possibility of fraud, it generates an alert. The input is the result of the fraud possibility assessment and the user's emotional state data, and the output is the alert content. The alert includes information about the possibility of fraud and the user's anxiety. Specifically, the alert content is "A potentially fraudulent conversation has been detected. The user is feeling anxious. Please check immediately."

[0768] Step 6:

[0769] The server sends the alert content to pre-registered contacts such as family members and the police. The input is the alert content, and the output is the sent alert. Specifically, alerts are sent via email or messaging apps.

[0770] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0771] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0772] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0773] [Third embodiment]

[0774] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0775] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0776] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0777] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0778] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0779] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0780] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0781] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0782] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0783] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0784] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0785] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0786] overview

[0787] This invention is a system that constantly monitors users' conversations to detect fraudulent billing in real time and sends alerts to family members and the police if there is a possibility of fraud. This system includes an analysis function using a generative AI model, and can make highly accurate judgments by comparing the results with a database of past fraud cases.

[0788] Configuring Users

[0789] Users install a dedicated app and perform the initial setup, registering contact information such as family members and the police. This allows the device to be set up so that alerts are sent to the registered contacts if fraud is detected.

[0790] Audio Capture and Conversion

[0791] The device is equipped with a microphone for voice capture, which records the user's conversation in real time. This voice data is converted into text using voice recognition technology. For example, if a user says, "Tell me the PIN number for my cash card," the voice data is instantly converted into text data.

[0792] Analysis and Evaluation

[0793] The converted text is sent from the device to a server, which uses a generative AI model to analyze the text and assess its likelihood of fraud. The generative AI model then compares the text with a database of past fraud cases to determine whether it indicates fraud.

[0794] Sending alerts

[0795] If a potential fraudulent activity is detected, the server will send an alert to registered contacts. The alert will include the content of the potentially fraudulent conversation and a message urging immediate action. For example, a notification might say, "A potentially fraudulent conversation has been detected. Please check immediately."

[0796] Specific examples

[0797] 1. User comments

[0798] User: "What's my cash card PIN?"

[0799] The device captures this audio and converts it into text.

[0800] Converted text: "What is your cash card PIN?"

[0801] 2. Server Analysis

[0802] The text is sent to a server and analyzed by a generative AI model.

[0803] The model compares the data with past fraud cases and rates the likelihood of fraud highly.

[0804] 3. Sending alerts

[0805] The server detects potential fraud and sends alerts to registered family members and the police.

[0806] Alert: "A potentially fraudulent conversation has been detected. Please investigate immediately."

[0807] merit

[0808] This system enables real-time detection of fraudulent billing scams and rapid response, providing effective crime prevention measures, particularly for individuals at high risk of fraud, such as the elderly. Furthermore, the use of generative AI models can improve the accuracy of fraud detection and reduce false positives.

[0809] The processing flow will be explained below.

[0810] Step 1:

[0811] Users install the app, launch it, and perform the initial setup, which includes registering contact information for family members, police, etc. This sets up where alerts will be sent if fraud is detected.

[0812] Step 2:

[0813] The terminal uses a microphone for voice capture to record the user's conversation in real time. This voice data is immediately sent to a voice recognition system by a processing unit in the terminal and converted into text format. For example, if a user says, "Tell me the PIN number for my cash card," this voice is converted into text data, "Tell me the PIN number for my cash card."

[0814] Step 3:

[0815] The converted text data is sent from the device to a server, which receives the text data and begins analyzing it using a generative AI model.

[0816] Step 4:

[0817] The server analyzes the received text data using a generative AI model to assess the likelihood of fraudulent activity, which then matches the text with past fraud cases stored in a database and runs algorithms to detect signs of fraud.

[0818] Step 5:

[0819] If the generative AI model's analysis determines that fraud is likely, the server generates an alert, which includes a description of the potential fraud and the conversation.

[0820] Step 6:

[0821] The server generates an alert and sends it to pre-registered contacts, including the user's family and the police, with a message urging them to immediately investigate the fraudulent activity, such as "A potentially fraudulent conversation has been detected. Please investigate immediately."

[0822] Step 7:

[0823] After an alert is sent, the server logs the results, including the fraudulent activity detected, the alert content, and the destination information, for later analysis and auditing.

[0824] Step 8:

[0825] Users who receive the alert can work with their family or the police to take action to prevent the fraud from progressing, for example by taking specific measures such as verifying personal information with callers before providing it.

[0826] This process flow protects users from fraudulent billing and allows for a quick response.

[0827] Example 1

[0828] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0829] Fraudulent activities, such as fictitious billing scams, have a serious impact, especially on vulnerable individuals such as the elderly. To address this issue, a system that can detect fraud in real time and respond quickly is needed. However, conventional methods have low accuracy in detecting fraud, and are prone to false positives and delays. Therefore, a method is needed to detect fraud in real time with high accuracy and respond quickly.

[0830] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0831] In this invention, the server includes a means for capturing a user's voice in real time and converting it into text, a means for analyzing the text using a generative AI model to evaluate the likelihood of fraud, a means for sending an alert to pre-registered contacts if a likelihood of fraud is detected, a means for comparing the text with a database of past fraud cases, and a means for sending the text to the server using encrypted communication, thereby enabling fraud to be detected with high accuracy in real time and alerting family members and the police quickly.

[0832] "Means for capturing and converting a user's voice into text in real time" refers to a device or software that instantly records the voice data spoken by a user and converts it into text data using voice recognition technology.

[0833] "Using a generative AI model to analyze said text and assess the likelihood of fraud" refers to technology that uses a machine learning model to analyze text data and determine the likelihood of fraud based on specific patterns or characteristics of fraud.

[0834] "Means for sending alerts to pre-defined contacts when potential fraud is detected" refers to a method or device for sending a warning message to pre-defined family or law enforcement contacts when signs of fraud are detected.

[0835] "Means for comparing the text with a database of past fraud cases" refers to technology for determining fraudulent activity by referring to a database that records past fraud cases and comparing and matching the text data to be analyzed.

[0836] "Means for transmitting the text to the server using encrypted communication" refers to a method that uses data encryption technology to securely transmit text data to the server.

[0837] overview

[0838] This invention is a system that detects fraudulent billing in real time by constantly monitoring the user's voice and sends an alert to family members and the police if there is a possibility of fraud. This system performs analysis using a generative AI model and compares the results with a database of past fraud cases, enabling highly accurate judgments.

[0839] System Configuration

[0840] This system is realized mainly using the following hardware and software.

[0841] Hardware

[0842] Device: A smartphone or tablet with a microphone for audio capture.

[0843] Server: A high-performance computer for running generative AI models.

[0844] software

[0845] Generative AI model: A machine learning model that analyzes fraudulent activity.

[0846] Speech recognition software: For example, using the Google Speech-to-Text API.

[0847] Database: A database containing past fraud cases.

[0848] Specific processing of the program

[0849] User Preferences

[0850] Users install a dedicated app on their device and register contact information for family members and the police, so that if fraud is detected, an alert is automatically sent to the designated contacts.

[0851] Audio capture and transcription

[0852] The device's built-in microphone constantly records the user's speech and converts this voice data into text data in real time. For example, if a user says, "Tell me my cash card PIN," the speech is converted into text using voice recognition software.

[0853] Sending and analyzing text data

[0854] The converted text data is sent to a server using encrypted communications. The server then analyzes the received text data using a generative AI model to assess the likelihood of fraud. This analysis includes a process of comparing it with a database of past fraud cases.

[0855] Sending alerts

[0856] If a fraudulent activity is detected, the server will send an alert to registered family members or the police. The alert will include the details of the potentially fraudulent conversation and a message urging immediate action. For example, a notification such as "A potentially fraudulent conversation has been detected. Please check immediately."

[0857] Examples and prompts

[0858] Specific examples

[0859] 1. User utterance: The user says, "Tell me the PIN number for my cash card."

[0860] What the device does: It captures this audio and converts it to text using speech recognition software.

[0861] Converted text: "What is your cash card PIN?"

[0862] 2. Server analysis: The converted text is sent to the server and analyzed using the generative AI model.

[0863] Analysis results: The generative AI model compares the data with a database of past fraud cases and evaluates the likelihood of fraudulent activity.

[0864] 3. Send alert: The server determines that there is a high possibility of fraud and sends an alert to pre-registered family members and the police.

[0865] Alert: "A potentially fraudulent conversation has been detected. Please investigate immediately."

[0866] Prompt Sentence Examples

[0867] To perform analysis, enter the following prompt into the generative AI model:

[0868] "Judge whether the following text is likely to be fraudulent: 'Tell me your bank card PIN'"

[0869] As described above, the present invention is a system that aims to analyze user conversations in real time, detect fraudulent acts with high accuracy, and respond quickly.

[0870] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0871] Step 1: Initial User Setup

[0872] Users install a dedicated app on their device. When they launch the app for the first time, they register contact information for family members and the police, which allows the app to send prompt notifications when fraud is detected. The input is the contact information entered by the user, and the output is that it is saved on the server.

[0873] Input: Family and police contact information

[0874] Output: Contact information is saved on the server

[0875] Specific operation: The user opens the app, enters the required information on the contact registration screen, and presses the save button.

[0876] Step 2: Capture audio

[0877] The device uses a built-in microphone to capture the user's speech in real time. The captured voice data is temporarily stored on the device. The input is the user's voice and the output is the captured voice data.

[0878] Input: User spoken words

[0879] Output: Captured audio data

[0880] What it does: The device runs the app in the background, and the microphone constantly captures audio and stores it in temporary storage.

[0881] Step 3: Convert audio data to text

[0882] The device converts the captured voice data into text data in real time using voice recognition software (e.g., Google Speech-to-Text API). The input is the captured voice data, and the output is text data.

[0883] Input: Captured audio data

[0884] Output: Converted text data

[0885] Specific operation: The saved voice data is converted into text data by voice recognition software.

[0886] Step 4: Sending text data

[0887] The terminal sends the converted text data to the server using encrypted communication. The input is the converted text data, and the output is the data sent to the server.

[0888] Input: Converted text data

[0889] Output: Text data is sent to the server

[0890] Specific operation: Once text data is generated, it is automatically sent to the server, and SSL / TLS encrypted communication is used.

[0891] Step 5: Analyzing the text data

[0892] The server analyzes the received text data using a generative AI model to assess the likelihood of fraud. The generative AI model makes its judgment based on previously learned fraud patterns. The input is the text data sent to the server, and the output is the assessment result of fraud.

[0893] Input: Text data sent to the server

[0894] Output: Fraud assessment results

[0895] Specific operation: The server inputs the received text data into a generative AI model, which then analyzes the data by comparing it with a database of past fraud cases.

[0896] Step 6: Generate fraud findings

[0897] The server receives the analysis results from the generative AI model and evaluates whether the likelihood of fraud is high or low. If it is judged to be high, it is considered fraud. The input is the analyzed text data, and the output is the fraud judgment result.

[0898] Input: Parsed text data

[0899] Output: Fraud detection result

[0900] Specific operation: The server receives the analysis evaluation results and identifies the possibility of fraud using a number or tag.

[0901] Step 7: Sending an alert

[0902] If the server determines that there is a high possibility of fraud, it will send an alert to registered family members or the police. The input is the fraud judgment result and pre-registered contact information, and the output is an alert message.

[0903] Input: Fraud detection result, registered contact information

[0904] Output: Alert message

[0905] What happens: The server generates an alert message and sends it via SMS, email, or in-app notification. Notification content: "A potentially fraudulent conversation has been detected. Please check immediately."

[0906] (Application example 1)

[0907] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0908] In recent years, there has been a sharp increase in fraudulent activities such as fictitious billing scams and bank transfer scams, with elderly people and those in socially vulnerable positions being particularly targeted. Conventional fraud prevention measures are limited to crime prevention education and warnings, and there are limitations to detecting fraud in real time and responding quickly, making it difficult to quickly prevent fraud damage, so a solution is needed.

[0909] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0910] In this invention, the server includes means for capturing a user's voice in real time and converting it into text, means for analyzing the text using a generative AI model to evaluate the possibility of fraud, means for sending an alert to pre-registered contacts when a possibility of fraud is detected, means for being installed on the smartphone and monitoring the user's calls in real time, means for recording the user's conversations and analyzing the content using a generative AI model, means for comparing the content with a database of past fraud cases to evaluate signs of fraud, and communication means for instantly sending the alert, thereby enabling fraud detection in real time and rapid response.

[0911] "Real-time voice capture" refers to technology that instantly records the voice spoken by the user and acquires the voice data.

[0912] "Convert to text" refers to the process of converting audio data into written information.

[0913] A "generative AI model" is an artificial intelligence algorithm or machine learning model that has been trained for a specific task.

[0914] "Analysis" is the process of evaluating and understanding data to understand its meaning and content.

[0915] "Fraudulent Potential" refers to an indicator or assessment of the likelihood that an action or statement may be fraudulent.

[0916] "Alert" means a notice or warning of an emergency or caution.

[0917] "Pre-registered contacts" refers to the phone numbers and email addresses that the user has set up in advance to receive notifications.

[0918] A "smartphone" refers to a mobile information terminal that has communication capabilities, internet connectivity, and can run applications.

[0919] "Real-time call monitoring" refers to the process of instantly monitoring and tracking user calls and communications.

[0920] "Recording audio" refers to the act of saving a user's speech or conversation on a recording medium.

[0921] "Content analysis" refers to the process of examining recorded data to understand its meaning and intent.

[0922] A "database of past fraud cases" refers to a collection of information that has accumulated data on fraud cases that have occurred in the past.

[0923] "Assessing for fraudulent indicators" refers to the process of determining the likelihood that a particular action or statement may be fraudulent and drawing a consequence.

[0924] "Communication means" refers to the technology and infrastructure for transmitting information, including the Internet and mobile communications.

[0925] System configuration

[0926] The present invention is implemented as a security application installed on a smartphone, which monitors users' calls in real time and performs a series of processes to detect fraudulent activity.

[0927] Hardware and software used

[0928] Smartphone: microphone, internet connection

[0929] Speech recognition software: Google Cloud Speech-to-Text API

[0930] Generative AI model: OpenAI GPT-4

[0931] Database: Database of past fraud cases

[0932] Communication Method: Uses Twilio API to send alerts via SMS and phone

[0933] Data processing and calculation

[0934] Audio Capture

[0935] The user's call is recorded in real time through the smartphone's microphone, and the audio data is captured in high quality and is used directly for the next step, which is to convert it into text.

[0936] Speech-to-text

[0937] The recorded voice data is converted into text data in real time using the Google Cloud Speech-to-Text API, so the words spoken by the user are recorded as text and can be used directly in subsequent analysis processes.

[0938] Text analytics

[0939] The generative AI model (OpenAI GPT-4) uses text data to analyze the content of utterances, using a pre-filled prompt, "Is this likely to be fraud?", to accurately assess the likelihood of fraud.

[0940] Comparison with a database of past fraud cases

[0941] The analyzed text data is compared with a database of past fraud cases, which contains a large number of past fraud methods and cases, and based on this, the system evaluates signs of fraud with high accuracy.

[0942] Sending alerts

[0943] If a fraudulent activity is deemed likely, an alert will be sent immediately to the user's pre-registered contacts (family, police, etc.) using the Twilio API. The alert will include the contents of the detected conversation and a message urging immediate action.

[0944] Specific examples

[0945] 1. Audio Capture

[0946] A user says on the phone, "Tell me the PIN number for my cash card."

[0947] The smartphone's microphone records this audio.

[0948] 2. Text Conversion

[0949] The recorded voice is converted into "Please tell me my cash card PIN number" via the Google Cloud Speech-to-Text API.

[0950] 3. Text Analysis

[0951] A generative AI model (OpenAI GPT-4) analyzes the text and evaluates it against the prompt, "Is this a possible scam?"

[0952] Evaluation result: "High probability of fraud"

[0953] 4. Sending alerts

[0954] Alerts are instantly sent to the user's family and police using the Twilio API.

[0955] Alert: "A potentially fraudulent conversation has been detected. Please investigate immediately."

[0956] Prompt Sentence Examples

[0957] text

[0958] Fraud prevention system:

[0959] Text generated from user's call data: "What is my cash card PIN?"

[0960] Prompt: "Could this be a scam?"

[0961] In this way, the present invention is able to detect fraudulent activity in real time and respond quickly.

[0962] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0963] Step 1:

[0964] A user initiates a call through a smartphone. The user's speech is captured in real time by the smartphone's microphone and collected as audio data. The input is the user's speech, and the output is the recorded audio data.

[0965] Step 2:

[0966] The device sends the captured audio data to the Google Cloud Speech-to-Text API, where it is analyzed and converted to text. The input is the original audio data, and the output is the converted text data.

[0967] Step 3:

[0968] The device sends the converted text data to a generative AI model (OpenAI GPT-4), which analyzes the text data using a pre-defined prompt: "Is this likely to be fraud?" The input is the text data and the prompt, and the output is an assessment of the likelihood of fraud.

[0969] Step 4:

[0970] The server receives the evaluation results from the generative AI model and compares them with a database of past fraud cases, where patterns and features that match fraudulent activity are further evaluated. The input is the text analysis results and database information, and the output is the final evaluation result.

[0971] Step 5:

[0972] If fraud is determined to be likely, the server uses the Twilio API to send an alert to pre-registered contacts, where the input is the evaluation result and registered contact information, and the output is the sent alert message.

[0973] Step 6:

[0974] Finally, the user's family or the police receive the alert and quickly confirm and take necessary action. In this step, the input is the alert message and the output is the confirmation of the message and the corresponding action.

[0975] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0976] overview

[0977] This invention is a system that constantly monitors users' conversations and emotions to detect fraudulent billing scams in real time, and sends alerts to family members and the police if there is a possibility of fraud. This system uses a generative AI model and an emotion engine to predict and detect fraud damage with high accuracy.

[0978] Configuring Users

[0979] Users install the app and perform the initial setup, which includes registering contact information for family members, police, etc. This sets up where alerts will be sent if fraud is detected.

[0980] Audio Capture and Conversion

[0981] The device is equipped with a microphone for voice capture, which records the user's conversation in real time. This voice data is converted into text using voice recognition technology. For example, if a user says, "Tell me the PIN number for my cash card," this voice is converted into text data: "Tell me the PIN number for my cash card."

[0982] emotion recognition

[0983] The converted voice data is analyzed by the device's emotion engine, which detects the user's emotional state from the voice data. This analysis evaluates whether the user is feeling stressed, relaxed, anxious, etc.

[0984] Analysis and Evaluation

[0985] The converted text data and the results of the sentiment analysis are sent from the device to a server. The server uses a generative AI model to analyze the text data and evaluate the likelihood of fraud. The generative AI model compares the text with past fraud cases stored in a database to detect signs of fraud. The user's emotional state also influences the evaluation.

[0986] Sending alerts

[0987] If the generative AI model's analysis determines that fraud is highly likely, the server generates an alert. This alert includes information about the potential fraud, the content of the conversation, and the user's emotional state. The alert is sent to pre-registered contacts (family members or police). If the user's emotional state is unstable, the alert's priority is increased.

[0988] Specific examples

[0989] 1. User statements and sentiment

[0990] User: "What's my cash card PIN?"

[0991] The device captures this audio and converts it into text.

[0992] The emotion engine analyzes the voice data and assesses that the user is feeling anxious.

[0993] 2. Server Analysis

[0994] The text and sentiment analysis results are sent to a server and analyzed by a generative AI model.

[0995] The model compares the data with past fraud cases and assesses the likelihood of fraud.

[0996] 3. Sending alerts

[0997] The server detects potential fraud and sends alerts to registered family members and the police.

[0998] Alert: "A potentially fraudulent conversation has been detected. The user is concerned. Please check immediately."

[0999] merit

[1000] This system enables real-time detection of fraudulent billing scams and rapid response. It can provide effective crime prevention measures, especially for individuals at high risk of becoming a victim of fraud, such as the elderly. Furthermore, by using an emotion engine, highly accurate fraud detection can be achieved by taking into account the user's mental state. This reduces false positives and improves the quality of responses.

[1001] The processing flow will be explained below.

[1002] Step 1:

[1003] Users install the app, launch it, and perform the initial setup, which includes registering contact information for family members, police, etc. This sets up where alerts will be sent if fraud is detected.

[1004] Step 2:

[1005] The device uses a microphone for voice capture to record the user's conversation in real time. The recorded voice data is processed within the device and converted into text data by a voice recognition system. For example, if a user says, "Tell me the PIN number for my cash card," this voice becomes the text data "Tell me the PIN number for my cash card."

[1006] Step 3:

[1007] The converted voice data is simultaneously analyzed by the device's emotion engine, which evaluates the user's emotional state from the voice data and determines whether the user is stressed, relaxed, or anxious. For example, if the user is assessed as being anxious, the level of anxiety is also recorded.

[1008] Step 4:

[1009] The converted text data and sentiment analysis results are sent from the device to a server. The server receives the text data and sentiment data and uses a generative AI model to analyze the text and assess the likelihood of fraud. The generative AI model compares the text with past fraud cases stored in a database to determine signs of fraud.

[1010] Step 5:

[1011] If the generative AI model's analysis determines that fraud is highly likely, the server generates an alert. This alert includes a statement that fraud is likely, as well as the user's comments and emotional state. For example, the alert could include the following: "A potentially fraudulent conversation has been detected. The user is feeling uneasy. Please check immediately."

[1012] Step 6:

[1013] The server generates an alert and sends it to pre-registered contacts (family members or the police). This alert allows the contacts to quickly identify possible fraud and take appropriate action. If the emotional state is unstable, the server sends an alert with a higher priority, encouraging a quicker response.

[1014] Step 7:

[1015] After an alert is sent, the server logs the results, including the fraudulent activity detected, the alert content, and the destination information, for later analysis and auditing.

[1016] Step 8:

[1017] Users who receive the alert can work with family members or law enforcement to take action to prevent the fraud from progressing, for example by reviewing the content of the call and taking specific steps to rule out the possibility of fraud.

[1018] Through this detailed process flow, users are protected from fraudulent billing scams with high accuracy and can respond quickly and effectively. The addition of an emotion engine enables advanced fraud detection that takes into account the user's mental state, improving overall safety and reliability.

[1019] Example 2

[1020] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1021] In today's world, elderly people and those unfamiliar with technology are increasingly being targeted by fraudulent billing scams. This puts many people at risk of financial loss. However, current security systems have difficulty detecting fraudulent activity in real time and taking appropriate action. Furthermore, because they do not take emotional states into account, they are prone to false positives and limited in the quality of the response. There is a need to solve these problems.

[1022] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1023] In this invention, the server includes: means for allowing a user to perform initial setup and register contact information; means for capturing the user's voice in real time and converting it into text; means for detecting the user's emotional state from the voice data and text data; means for transmitting the converted text data and emotion analysis results to the server; means for analyzing the text using a generative AI model and evaluating the possibility of fraud; and means for sending an alert to pre-registered contacts if the possibility of fraud is detected. This makes it possible to monitor user conversations in real time, detect fraud risks with high accuracy, and respond quickly.

[1024] "Means for users to set up and register contact information" refers to the process by which, after installing the application, users enter contact information for themselves and related parties and store it in a local database for subsequent data transmission and notifications.

[1025] "Means for capturing the user's voice in real time and converting it into text" refers to a function that uses the device's built-in microphone to record the user's speech and converts the voice data into a corresponding text format using voice recognition technology.

[1026] "Means for detecting a user's emotional state from voice data and text data" refers to technology that analyzes the content of a user's speech and voice characteristics to identify the emotions the user is feeling (e.g., anxiety, stress, relaxation).

[1027] "Means for transmitting the converted text data and emotion analysis results to the server" refers to a mechanism for transmitting the text data and emotion state acquired and analyzed on the device to the server using a secure communication protocol.

[1028] "Using a generative AI model to analyze the text and assess the likelihood of fraud" refers to the process of inputting received text data into an AI model based on a neural network or other advanced algorithm to assess the risk of fraud.

[1029] "Means of sending alerts to pre-registered contacts if possible fraudulent activity is detected" refers to a function that sends notifications to pre-registered emergency contacts (e.g., family members, police) if the evaluation results indicate a high possibility of fraudulent activity.

[1030] MODE FOR CARRYING OUT THE INVENTION

[1031] This invention is a system that constantly monitors users' conversations and emotions to detect fraudulent billing scams in real time, and sends alerts to family members and the police if there is a possibility of fraud. This system uses a generative AI model and an emotion engine to predict and detect fraud damage with high accuracy.

[1032] Installing and setting up the dedicated app

[1033] The user first installs the dedicated app and performs the initial setup, which includes registering the user's own contact information as well as emergency contact information for family members, police, etc. At this stage, the registered contact information is securely stored in the device's local database. The information entered by the user will be used for subsequent data transmission and alert generation.

[1034] Voice capture and text transcription

[1035] The device is equipped with a microphone for voice capture, which records the user's conversation in real time. This voice data is converted into text using voice recognition technology. A specific technology that can be used is Google Cloud Speech-to-Text. For example, if a user says, "Tell me the PIN number for my cash card," this voice is converted into text data: "Tell me the PIN number for my cash card."

[1036] emotion recognition

[1037] The converted voice data is analyzed by the device's emotion engine. A specific example of an emotion engine is IBM Watson's Natural Language Understanding, which can identify the user's emotional state (e.g., anxiety, stress, relaxation, etc.) from the content of the user's speech and voice characteristics. For example, the text and voice of a statement such as "Tell me my cash card PIN number" are analyzed and the user is evaluated as feeling anxious.

[1038] Sending data to the server

[1039] The sentiment analysis results and text data are sent to the server using a secure communication protocol such as HTTPS. The data sent includes the converted text data and the emotional state, allowing even sensitive information to be handled safely.

[1040] Potential fraud analysis

[1041] The server feeds the received data into a generative AI model (e.g., GPT-4) to assess the likelihood of fraud, for example using the following prompt:

[1042] "Analyze the following conversation and assess the likelihood of fraud. The user's emotion is 'anxiety'. Conversation: 'Please tell me your cash card PIN number.'"

[1043] The analysis results obtained in this way are compared with a database of past fraud cases to detect signs of fraud.

[1044] Generate and send alerts

[1045] If the generative AI model's analysis determines that fraud is likely, the server generates an alert. This alert includes information about the potential fraud, the content of the conversation, and the user's emotional state. The alert is sent to pre-registered contacts (e.g., family, police). The alert notification may include a message such as:

[1046] "A potentially fraudulent conversation has been detected. The user is concerned. Please check immediately."

[1047] This system monitors user conversations in real time, detects fraud risks with high accuracy, and responds quickly. It can provide effective crime prevention measures, especially for individuals at high risk of becoming a victim of fraud, such as the elderly. By using an emotion engine, highly accurate fraud detection is achieved that takes into account the user's mental state, reducing false positives and improving the quality of responses.

[1048] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1049] Step 1: Install and set up the dedicated app

[1050] The user installs the app. After installation, the app displays an initial setup screen where the user enters contact information for themselves, their family, the police, etc. The entered contact information is saved in the device's local database. The entered data is used in subsequent processing of the app.

[1051] Input: User's contact information

[1052] Output: Contact information stored in a local database

[1053] Specific behavior:

[1054] The user installs the app and accesses the initial setup screen.

[1055] You will be prompted to enter contact information for your family members.

[1056] When the user enters contact information and presses the "Save" button, the contact information is saved to a local database.

[1057] Step 2: Capture audio and convert it to text

[1058] The device uses a built-in microphone to record the user's speech in real time, and this voice data is converted to text using the Google Cloud Speech-to-Text API.

[1059] Input: User's voice data

[1060] Output: Text data

[1061] Specific behavior:

[1062] When the user starts speaking, the device's microphone starts capturing it.

[1063] For example, voice data saying "Tell me the PIN number for my cash card" is acquired.

[1064] The voice data is passed to the Google Cloud Speech-to-Text API and converted into text: "What is my cash card PIN?"

[1065] Step 3: Emotion Recognition

[1066] The device passes the converted text data to an emotion engine that uses IBM Watson's Natural Language Understanding to detect the user's emotional state from the voice and text data.

[1067] Input: Text data and audio data

[1068] Output: User's emotional state

[1069] Specific behavior:

[1070] The text data "Please tell me the PIN number for my cash card" is input to the emotion engine.

[1071] The emotion engine analyzes the tone and content of the user's voice.

[1072] As a result of the analysis, it is detected that the user is feeling "anxiety."

[1073] Step 4: Sending data to the server

[1074] The device sends the converted text data and the emotion analysis results to the server using the HTTP protocol.

[1075] Input: Text data and sentiment analysis results

[1076] Output: Data sent to the server

[1077] Specific behavior:

[1078] The text data generated on the device, "Please tell me your cash card PIN number" and the emotional state "anxiety," are structured in JSON format.

[1079] This data is sent to the server using the HTTPS protocol.

[1080] Step 5: Fraud Potential Analysis

[1081] The server inputs the received data into a generative AI model (e.g., GPT-4), which compares it with a database of past fraud cases to assess the likelihood of fraud.

[1082] Input: Text data, sentiment analysis results

[1083] Output: Fraud likelihood assessment result

[1084] Specific behavior:

[1085] The server generates a prompt. For example, "Analyze the following conversation and assess the likelihood of fraud. The user's emotion is detected as 'anxiety.' Conversation: 'Please tell me your cash card PIN number.'"

[1086] Text data along with a prompt sentence is input into the generative AI model.

[1087] The generative AI model performs the analysis and returns a result that indicates there is a high probability of fraud.

[1088] Step 6: Generate and send an alert

[1089] If the server determines that fraud is likely, it generates an alert. This alert includes information about the potential fraud, the content of the conversation, and the user's emotional state. The alert is sent to pre-registered contacts (family members or the police).

[1090] Input: Fraud possibility assessment results, registered contact information

[1091] Output: Alert notification sent

[1092] Specific behavior:

[1093] The server generates an alert message saying, "A potentially fraudulent conversation has been detected. The user is concerned. Please check immediately."

[1094] Alerts will be sent via email and SMS based on the registered contact information.

[1095] This system makes it possible to monitor user conversations in real time, detect fraud risks with high accuracy, and respond quickly.

[1096] (Application example 2)

[1097] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1098] Elderly people and those living alone are at increased risk of falling victim to fraudulent billing scams. These fraudulent activities are becoming more sophisticated, making simple fraud detection systems insufficient. Furthermore, more accurate judgments are required by taking into account the user's emotional state, but current systems are lacking in this area. Therefore, the object of this invention is to provide a security system that constantly monitors the user's conversations and emotions, detects fraudulent activities in real time with high accuracy, and responds quickly and appropriately.

[1099] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing a user's voice in real time and converting it into text, means for analyzing the text using a generative AI model and evaluating the possibility of fraud, means for sending an alert to pre-registered contacts when the possibility of fraud is detected, means for evaluating the user's emotional state using an emotion analysis engine, and means for including the user's emotional state in the alert along with the possibility of fraud. This enables highly accurate fraud detection and rapid response that takes the user's mental state into consideration.

[1100] "Means for capturing and converting user speech into text in real time" refers to a combination of devices and algorithms for quickly capturing speech emitted by a user and converting it into text information.

[1101] "Means for analyzing said text using a generative AI model to assess the likelihood of fraudulent activity" refers to a system that uses AI technology to analyze text data and determine whether it is related to fraudulent activity.

[1102] "A means of sending alerts to pre-registered contacts when possible fraudulent activity is detected" is a function that sends an alert to pre-set contacts such as family members or the police when signs of fraud are detected.

[1103] "Means for assessing a user's emotional state using an emotion analysis engine" refers to technology that analyzes voice and text data to recognize and evaluate the emotions felt by the user.

[1104] The "means for including the user's emotional state along with the possibility of fraud in the alert" is a mechanism for including the user's emotional state in a warning message indicating the possibility of fraud.

[1105] This invention is a system that monitors user conversations and emotions in real time, detects fraudulent billing with high accuracy, and promptly sends an alert. Specific embodiments for implementing this invention will be described below.

[1106] 1. System Configuration

[1107] Audio capture device:

[1108] Devices such as smartphones have built-in microphones that are used to capture the user's voice in real time.

[1109] Speech-to-text methods:

[1110] The captured audio data is converted to text using the Google Cloud Speech-to-Text API, which makes the audio information available as text.

[1111] Analysis methods using generative AI models:

[1112] The converted text data is then analyzed using a generative AI model, such as OpenAI's GPT-3, which compares it with a database of past fraud cases to assess the likelihood of fraud.

[1113] Sentiment Analysis Engine:

[1114] It uses IBM Watson's emotion analysis engine to analyze the user's emotional state, for example, assessing the user's stress, anxiety, or level of relaxation from voice data.

[1115] Alerts sent via:

[1116] If a fraudulent activity is assessed as likely, an alert will be sent to pre-registered contacts such as family members or the police, and the alert will include information indicating the possibility of fraud and the user's emotional state.

[1117] 2. Specific examples of processing

[1118] Capturing user conversations and emotions

[1119] Suppose a user says into their smartphone, "What is my PIN number for my cash card?" The smartphone's microphone captures this speech and sends it in real time to the Google Cloud Speech-to-Text API, where it is converted into text.

[1120] Emotion Analysis

[1121] The converted text and voice data is sent to IBM Watson's sentiment analysis engine to assess whether the user is feeling anxious.

[1122] Fraud Analysis

[1123] A generative AI model (OpenAI GPT-3) analyzes the converted text data and compares it with a database of past fraud cases. For example, the following prompt sentence is used:

[1124] Example prompt sentence:

[1125] Please rate the likelihood that the following conversation text is a scam.

[1126] Text: What is your cash card PIN number?

[1127] If the generative AI model's analysis determines that there is a high possibility of fraud, the system will generate an alert.

[1128] Sending alerts

[1129] The alert will be sent to pre-registered family members and the police. The alert content will include information about the possibility of fraud and the user's concerns. For example, an alert may be sent with the content, "A potentially fraudulent conversation has been detected. The user is concerned. Please check immediately."

[1130] The system of the present invention can protect elderly people and those living alone from sophisticated fraud. Furthermore, by taking into account the user's emotional state, it is expected to achieve more accurate fraud detection and reduce false positives.

[1131] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1132] Step 1:

[1133] The user speaks into the smartphone. The smartphone's microphone captures the voice in real time. The input is voice data, and the output is the captured voice. For example, the user might record a voice saying, "Tell me my cash card PIN."

[1134] Step 2:

[1135] The smartphone sends the captured voice data to the Google Cloud Speech-to-Text API, which converts it into text data in real time. The input is the captured voice data, and the output is the text data corresponding to that voice. Specifically, the voice saying "Tell me my cash card PIN" is converted into text "Tell me my cash card PIN."

[1136] Step 3:

[1137] The converted text data and voice data are sent to IBM Watson's emotion analysis engine. The emotion analysis engine analyzes this data and evaluates the user's emotional state. The input is text data and voice data, and the output is analyzed emotional state data. In concrete terms, the text "Tell me my cash card PIN" is evaluated as indicating that the user is feeling anxious.

[1138] Step 4:

[1139] A generative AI model (OpenAI GPT-3) analyzes the converted text data. The input is the text data and the sentiment analysis results, and the output is an assessment of the likelihood of fraud. The prompt used is "Please assess the likelihood that the following conversational text is fraudulent. Text: Tell me your cash card PIN." The analysis results are obtained.

[1140] Step 5:

[1141] If the server determines that there is a high possibility of fraud, it generates an alert. The input is the result of the fraud possibility assessment and the user's emotional state data, and the output is the alert content. The alert includes information about the possibility of fraud and the user's anxiety. Specifically, the alert content is "A potentially fraudulent conversation has been detected. The user is feeling anxious. Please check immediately."

[1142] Step 6:

[1143] The server sends the alert content to pre-registered contacts such as family members and the police. The input is the alert content, and the output is the sent alert. Specifically, alerts are sent via email or messaging apps.

[1144] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1145] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1146] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1147] [Fourth embodiment]

[1148] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1149] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1150] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1151] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1152] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1153] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1154] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1155] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1156] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1157] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1158] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1159] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1160] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1161] overview

[1162] This invention is a system that constantly monitors users' conversations to detect fraudulent billing in real time and sends alerts to family members and the police if there is a possibility of fraud. This system includes an analysis function using a generative AI model, and can make highly accurate judgments by comparing the results with a database of past fraud cases.

[1163] Configuring Users

[1164] Users install a dedicated app and perform the initial setup, registering contact information such as family members and the police. This allows the device to be set up so that alerts are sent to the registered contacts if fraud is detected.

[1165] Audio Capture and Conversion

[1166] The device is equipped with a microphone for voice capture, which records the user's conversation in real time. This voice data is converted into text using voice recognition technology. For example, if a user says, "Tell me the PIN number for my cash card," the voice data is instantly converted into text data.

[1167] Analysis and Evaluation

[1168] The converted text is sent from the device to a server, which uses a generative AI model to analyze the text and assess its likelihood of fraud. The generative AI model then compares the text with a database of past fraud cases to determine whether it indicates fraud.

[1169] Sending alerts

[1170] If a potential fraudulent activity is detected, the server will send an alert to registered contacts. The alert will include the content of the potentially fraudulent conversation and a message urging immediate action. For example, a notification might say, "A potentially fraudulent conversation has been detected. Please check immediately."

[1171] Specific examples

[1172] 1. User comments

[1173] User: "What's my cash card PIN?"

[1174] The device captures this audio and converts it into text.

[1175] Converted text: "What is your cash card PIN?"

[1176] 2. Server Analysis

[1177] The text is sent to a server and analyzed by a generative AI model.

[1178] The model compares the data with past fraud cases and rates the likelihood of fraud highly.

[1179] 3. Sending alerts

[1180] The server detects potential fraud and sends alerts to registered family members and the police.

[1181] Alert: "A potentially fraudulent conversation has been detected. Please investigate immediately."

[1182] merit

[1183] This system enables real-time detection of fraudulent billing scams and rapid response, providing effective crime prevention measures, particularly for individuals at high risk of fraud, such as the elderly. Furthermore, the use of generative AI models can improve the accuracy of fraud detection and reduce false positives.

[1184] The processing flow will be explained below.

[1185] Step 1:

[1186] Users install the app, launch it, and perform the initial setup, which includes registering contact information for family members, police, etc. This sets up where alerts will be sent if fraud is detected.

[1187] Step 2:

[1188] The terminal uses a microphone for voice capture to record the user's conversation in real time. This voice data is immediately sent to a voice recognition system by a processing unit in the terminal and converted into text format. For example, if a user says, "Tell me the PIN number for my cash card," this voice is converted into text data, "Tell me the PIN number for my cash card."

[1189] Step 3:

[1190] The converted text data is sent from the device to a server, which receives the text data and begins analyzing it using a generative AI model.

[1191] Step 4:

[1192] The server analyzes the received text data using a generative AI model to assess the likelihood of fraudulent activity, which then matches the text with past fraud cases stored in a database and runs algorithms to detect signs of fraud.

[1193] Step 5:

[1194] If the generative AI model's analysis determines that fraud is likely, the server generates an alert, which includes a description of the potential fraud and the conversation.

[1195] Step 6:

[1196] The server generates an alert and sends it to pre-registered contacts, including the user's family and the police, with a message urging them to immediately investigate the fraudulent activity, such as "A potentially fraudulent conversation has been detected. Please investigate immediately."

[1197] Step 7:

[1198] After an alert is sent, the server logs the results, including the fraudulent activity detected, the alert content, and the destination information, for later analysis and auditing.

[1199] Step 8:

[1200] Users who receive the alert can work with their family or the police to take action to prevent the fraud from progressing, for example by taking specific measures such as verifying personal information with callers before providing it.

[1201] This process flow protects users from fraudulent billing and allows for a quick response.

[1202] Example 1

[1203] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1204] Fraudulent activities, such as fictitious billing scams, have a serious impact, especially on vulnerable individuals such as the elderly. To address this issue, a system that can detect fraud in real time and respond quickly is needed. However, conventional methods have low accuracy in detecting fraud, and are prone to false positives and delays. Therefore, a method is needed to detect fraud in real time with high accuracy and respond quickly.

[1205] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1206] In this invention, the server includes a means for capturing a user's voice in real time and converting it into text, a means for analyzing the text using a generative AI model to evaluate the likelihood of fraud, a means for sending an alert to pre-registered contacts if a likelihood of fraud is detected, a means for comparing the text with a database of past fraud cases, and a means for sending the text to the server using encrypted communication, thereby enabling fraud to be detected with high accuracy in real time and alerting family members and the police quickly.

[1207] "Means for capturing and converting a user's voice into text in real time" refers to a device or software that instantly records the voice data spoken by a user and converts it into text data using voice recognition technology.

[1208] "Using a generative AI model to analyze said text and assess the likelihood of fraud" refers to technology that uses a machine learning model to analyze text data and determine the likelihood of fraud based on specific patterns or characteristics of fraud.

[1209] "Means for sending alerts to pre-defined contacts when potential fraud is detected" refers to a method or device for sending a warning message to pre-defined family or law enforcement contacts when signs of fraud are detected.

[1210] "Means for comparing the text with a database of past fraud cases" refers to technology for determining fraudulent activity by referring to a database that records past fraud cases and comparing and matching the text data to be analyzed.

[1211] "Means for transmitting the text to the server using encrypted communication" refers to a method that uses data encryption technology to securely transmit text data to the server.

[1212] overview

[1213] This invention is a system that detects fraudulent billing in real time by constantly monitoring the user's voice and sends an alert to family members and the police if there is a possibility of fraud. This system performs analysis using a generative AI model and compares the results with a database of past fraud cases, enabling highly accurate judgments.

[1214] System Configuration

[1215] This system is realized mainly using the following hardware and software.

[1216] Hardware

[1217] Device: A smartphone or tablet with a microphone for audio capture.

[1218] Server: A high-performance computer for running generative AI models.

[1219] software

[1220] Generative AI model: A machine learning model that analyzes fraudulent activity.

[1221] Speech recognition software: For example, using the Google Speech-to-Text API.

[1222] Database: A database containing past fraud cases.

[1223] Specific processing of the program

[1224] User Preferences

[1225] Users install a dedicated app on their device and register contact information for family members and the police, so that if fraud is detected, an alert is automatically sent to the designated contacts.

[1226] Audio capture and transcription

[1227] The device's built-in microphone constantly records the user's speech and converts this voice data into text data in real time. For example, if a user says, "Tell me my cash card PIN," the speech is converted into text using voice recognition software.

[1228] Sending and analyzing text data

[1229] The converted text data is sent to a server using encrypted communications. The server then analyzes the received text data using a generative AI model to assess the likelihood of fraud. This analysis includes a process of comparing it with a database of past fraud cases.

[1230] Sending alerts

[1231] If a fraudulent activity is detected, the server will send an alert to registered family members or the police. The alert will include the details of the potentially fraudulent conversation and a message urging immediate action. For example, a notification such as "A potentially fraudulent conversation has been detected. Please check immediately."

[1232] Examples and prompts

[1233] Specific examples

[1234] 1. User utterance: The user says, "Tell me the PIN number for my cash card."

[1235] What the device does: It captures this audio and converts it to text using speech recognition software.

[1236] Converted text: "What is your cash card PIN?"

[1237] 2. Server analysis: The converted text is sent to the server and analyzed using the generative AI model.

[1238] Analysis results: The generative AI model compares the data with a database of past fraud cases and evaluates the likelihood of fraudulent activity.

[1239] 3. Send alert: The server determines that there is a high possibility of fraud and sends an alert to pre-registered family members and the police.

[1240] Alert: "A potentially fraudulent conversation has been detected. Please investigate immediately."

[1241] Prompt Sentence Examples

[1242] To perform analysis, enter the following prompt into the generative AI model:

[1243] "Judge whether the following text is likely to be fraudulent: 'Tell me your bank card PIN'"

[1244] As described above, the present invention is a system that aims to analyze user conversations in real time, detect fraudulent acts with high accuracy, and respond quickly.

[1245] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1246] Step 1: Initial User Setup

[1247] Users install a dedicated app on their device. When they launch the app for the first time, they register contact information for family members and the police, which allows the app to send prompt notifications when fraud is detected. The input is the contact information entered by the user, and the output is that it is saved on the server.

[1248] Input: Family and police contact information

[1249] Output: Contact information is saved on the server

[1250] Specific operation: The user opens the app, enters the required information on the contact registration screen, and presses the save button.

[1251] Step 2: Capture audio

[1252] The device uses a built-in microphone to capture the user's speech in real time. The captured voice data is temporarily stored on the device. The input is the user's voice and the output is the captured voice data.

[1253] Input: User spoken words

[1254] Output: Captured audio data

[1255] What it does: The device runs the app in the background, and the microphone constantly captures audio and stores it in temporary storage.

[1256] Step 3: Convert audio data to text

[1257] The device converts the captured voice data into text data in real time using voice recognition software (e.g., Google Speech-to-Text API). The input is the captured voice data, and the output is text data.

[1258] Input: Captured audio data

[1259] Output: Converted text data

[1260] Specific operation: The saved voice data is converted into text data by voice recognition software.

[1261] Step 4: Sending text data

[1262] The terminal sends the converted text data to the server using encrypted communication. The input is the converted text data, and the output is the data sent to the server.

[1263] Input: Converted text data

[1264] Output: Text data is sent to the server

[1265] Specific operation: Once text data is generated, it is automatically sent to the server, and SSL / TLS encrypted communication is used.

[1266] Step 5: Analyzing the text data

[1267] The server analyzes the received text data using a generative AI model to assess the likelihood of fraud. The generative AI model makes its judgment based on previously learned fraud patterns. The input is the text data sent to the server, and the output is the assessment result of fraud.

[1268] Input: Text data sent to the server

[1269] Output: Fraud assessment results

[1270] Specific operation: The server inputs the received text data into a generative AI model, which then analyzes the data by comparing it with a database of past fraud cases.

[1271] Step 6: Generate fraud findings

[1272] The server receives the analysis results from the generative AI model and evaluates whether the likelihood of fraud is high or low. If it is judged to be high, it is considered fraud. The input is the analyzed text data, and the output is the fraud judgment result.

[1273] Input: Parsed text data

[1274] Output: Fraud detection result

[1275] Specific operation: The server receives the analysis evaluation results and identifies the possibility of fraud using a number or tag.

[1276] Step 7: Sending an alert

[1277] If the server determines that there is a high possibility of fraud, it will send an alert to registered family members or the police. The input is the fraud judgment result and pre-registered contact information, and the output is an alert message.

[1278] Input: Fraud detection result, registered contact information

[1279] Output: Alert message

[1280] What happens: The server generates an alert message and sends it via SMS, email, or in-app notification. Notification content: "A potentially fraudulent conversation has been detected. Please check immediately."

[1281] (Application example 1)

[1282] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1283] In recent years, there has been a sharp increase in fraudulent activities such as fictitious billing scams and bank transfer scams, with elderly people and those in socially vulnerable positions being particularly targeted. Conventional fraud prevention measures are limited to crime prevention education and warnings, and there are limitations to detecting fraud in real time and responding quickly, making it difficult to quickly prevent fraud damage, so a solution is needed.

[1284] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1285] In this invention, the server includes means for capturing a user's voice in real time and converting it into text, means for analyzing the text using a generative AI model to evaluate the possibility of fraud, means for sending an alert to pre-registered contacts when a possibility of fraud is detected, means for being installed on the smartphone and monitoring the user's calls in real time, means for recording the user's conversations and analyzing the content using a generative AI model, means for comparing the content with a database of past fraud cases to evaluate signs of fraud, and communication means for instantly sending the alert, thereby enabling fraud detection in real time and rapid response.

[1286] "Real-time voice capture" refers to technology that instantly records the voice spoken by the user and acquires the voice data.

[1287] "Convert to text" refers to the process of converting audio data into written information.

[1288] A "generative AI model" is an artificial intelligence algorithm or machine learning model that has been trained for a specific task.

[1289] "Analysis" is the process of evaluating and understanding data to understand its meaning and content.

[1290] "Fraudulent Potential" refers to an indicator or assessment of the likelihood that an action or statement may be fraudulent.

[1291] "Alert" means a notice or warning of an emergency or caution.

[1292] "Pre-registered contacts" refers to the phone numbers and email addresses that the user has set up in advance to receive notifications.

[1293] A "smartphone" refers to a mobile information terminal that has communication capabilities, internet connectivity, and can run applications.

[1294] "Real-time call monitoring" refers to the process of instantly monitoring and tracking user calls and communications.

[1295] "Recording audio" refers to the act of saving a user's speech or conversation on a recording medium.

[1296] "Content analysis" refers to the process of examining recorded data to understand its meaning and intent.

[1297] A "database of past fraud cases" refers to a collection of information that has accumulated data on fraud cases that have occurred in the past.

[1298] "Assessing for fraudulent indicators" refers to the process of determining the likelihood that a particular action or statement may be fraudulent and drawing a consequence.

[1299] "Communication means" refers to the technology and infrastructure for transmitting information, including the Internet and mobile communications.

[1300] System configuration

[1301] The present invention is implemented as a security application installed on a smartphone, which monitors users' calls in real time and performs a series of processes to detect fraudulent activity.

[1302] Hardware and software used

[1303] Smartphone: microphone, internet connection

[1304] Speech recognition software: Google Cloud Speech-to-Text API

[1305] Generative AI model: OpenAI GPT-4

[1306] Database: Database of past fraud cases

[1307] Communication Method: Uses Twilio API to send alerts via SMS and phone

[1308] Data processing and calculation

[1309] Audio Capture

[1310] The user's call is recorded in real time through the smartphone's microphone, and the audio data is captured in high quality and is used directly for the next step, which is to convert it into text.

[1311] Speech-to-text

[1312] The recorded voice data is converted into text data in real time using the Google Cloud Speech-to-Text API, so the words spoken by the user are recorded as text and can be used directly in subsequent analysis processes.

[1313] Text analytics

[1314] The generative AI model (OpenAI GPT-4) uses text data to analyze the content of utterances, using a pre-filled prompt, "Is this likely to be fraud?", to accurately assess the likelihood of fraud.

[1315] Comparison with a database of past fraud cases

[1316] The analyzed text data is compared with a database of past fraud cases, which contains a large number of past fraud methods and cases, and based on this, the system evaluates signs of fraud with high accuracy.

[1317] Sending alerts

[1318] If a fraudulent activity is deemed likely, an alert will be sent immediately to the user's pre-registered contacts (family, police, etc.) using the Twilio API. The alert will include the contents of the detected conversation and a message urging immediate action.

[1319] Specific examples

[1320] 1. Audio Capture

[1321] A user says on the phone, "Tell me the PIN number for my cash card."

[1322] The smartphone's microphone records this audio.

[1323] 2. Text Conversion

[1324] The recorded voice is converted into "Please tell me my cash card PIN number" via the Google Cloud Speech-to-Text API.

[1325] 3. Text Analysis

[1326] A generative AI model (OpenAI GPT-4) analyzes the text and evaluates it against the prompt, "Is this a possible scam?"

[1327] Evaluation result: "High probability of fraud"

[1328] 4. Sending alerts

[1329] Alerts are instantly sent to the user's family and police using the Twilio API.

[1330] Alert: "A potentially fraudulent conversation has been detected. Please investigate immediately."

[1331] Prompt Sentence Examples

[1332] text

[1333] Fraud prevention system:

[1334] Text generated from user's call data: "What is my cash card PIN?"

[1335] Prompt: "Could this be a scam?"

[1336] In this way, the present invention is able to detect fraudulent activity in real time and respond quickly.

[1337] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1338] Step 1:

[1339] A user initiates a call through a smartphone. The user's speech is captured in real time by the smartphone's microphone and collected as audio data. The input is the user's speech, and the output is the recorded audio data.

[1340] Step 2:

[1341] The device sends the captured audio data to the Google Cloud Speech-to-Text API, where it is analyzed and converted to text. The input is the original audio data, and the output is the converted text data.

[1342] Step 3:

[1343] The device sends the converted text data to a generative AI model (OpenAI GPT-4), which analyzes the text data using a pre-defined prompt: "Is this likely to be fraud?" The input is the text data and the prompt, and the output is an assessment of the likelihood of fraud.

[1344] Step 4:

[1345] The server receives the evaluation results from the generative AI model and compares them with a database of past fraud cases, where patterns and features that match fraudulent activity are further evaluated. The input is the text analysis results and database information, and the output is the final evaluation result.

[1346] Step 5:

[1347] If fraud is determined to be likely, the server uses the Twilio API to send an alert to pre-registered contacts, where the input is the evaluation result and registered contact information, and the output is the sent alert message.

[1348] Step 6:

[1349] Finally, the user's family or the police receive the alert and quickly confirm and take necessary action. In this step, the input is the alert message and the output is the confirmation of the message and the corresponding action.

[1350] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1351] overview

[1352] This invention is a system that constantly monitors users' conversations and emotions to detect fraudulent billing scams in real time, and sends alerts to family members and the police if there is a possibility of fraud. This system uses a generative AI model and an emotion engine to predict and detect fraud damage with high accuracy.

[1353] Configuring Users

[1354] Users install the app and perform the initial setup, which includes registering contact information for family members, police, etc. This sets up where alerts will be sent if fraud is detected.

[1355] Audio Capture and Conversion

[1356] The device is equipped with a microphone for voice capture, which records the user's conversation in real time. This voice data is converted into text using voice recognition technology. For example, if a user says, "Tell me the PIN number for my cash card," this voice is converted into text data: "Tell me the PIN number for my cash card."

[1357] emotion recognition

[1358] The converted voice data is analyzed by the device's emotion engine, which detects the user's emotional state from the voice data. This analysis evaluates whether the user is feeling stressed, relaxed, anxious, etc.

[1359] Analysis and Evaluation

[1360] The converted text data and the results of the sentiment analysis are sent from the device to a server. The server uses a generative AI model to analyze the text data and evaluate the likelihood of fraud. The generative AI model compares the text with past fraud cases stored in a database to detect signs of fraud. The user's emotional state also influences the evaluation.

[1361] Sending alerts

[1362] If the generative AI model's analysis determines that fraud is highly likely, the server generates an alert. This alert includes information about the potential fraud, the content of the conversation, and the user's emotional state. The alert is sent to pre-registered contacts (family members or police). If the user's emotional state is unstable, the alert's priority is increased.

[1363] Specific examples

[1364] 1. User statements and sentiment

[1365] User: "What's my cash card PIN?"

[1366] The device captures this audio and converts it into text.

[1367] The emotion engine analyzes the voice data and assesses that the user is feeling anxious.

[1368] 2. Server Analysis

[1369] The text and sentiment analysis results are sent to a server and analyzed by a generative AI model.

[1370] The model compares the data with past fraud cases and assesses the likelihood of fraud.

[1371] 3. Sending alerts

[1372] The server detects potential fraud and sends alerts to registered family members and the police.

[1373] Alert: "A potentially fraudulent conversation has been detected. The user is concerned. Please check immediately."

[1374] merit

[1375] This system enables real-time detection of fraudulent billing scams and rapid response. It can provide effective crime prevention measures, especially for individuals at high risk of becoming a victim of fraud, such as the elderly. Furthermore, by using an emotion engine, highly accurate fraud detection can be achieved by taking into account the user's mental state. This reduces false positives and improves the quality of responses.

[1376] The processing flow will be explained below.

[1377] Step 1:

[1378] Users install the app, launch it, and perform the initial setup, which includes registering contact information for family members, police, etc. This sets up where alerts will be sent if fraud is detected.

[1379] Step 2:

[1380] The device uses a microphone for voice capture to record the user's conversation in real time. The recorded voice data is processed within the device and converted into text data by a voice recognition system. For example, if a user says, "Tell me the PIN number for my cash card," this voice becomes the text data "Tell me the PIN number for my cash card."

[1381] Step 3:

[1382] The converted voice data is simultaneously analyzed by the device's emotion engine, which evaluates the user's emotional state from the voice data and determines whether the user is stressed, relaxed, or anxious. For example, if the user is assessed as being anxious, the level of anxiety is also recorded.

[1383] Step 4:

[1384] The converted text data and sentiment analysis results are sent from the device to a server. The server receives the text data and sentiment data and uses a generative AI model to analyze the text and assess the likelihood of fraud. The generative AI model compares the text with past fraud cases stored in a database to determine signs of fraud.

[1385] Step 5:

[1386] If the generative AI model's analysis determines that fraud is highly likely, the server generates an alert. This alert includes a statement that fraud is likely, as well as the user's comments and emotional state. For example, the alert could include the following: "A potentially fraudulent conversation has been detected. The user is feeling uneasy. Please check immediately."

[1387] Step 6:

[1388] The server generates an alert and sends it to pre-registered contacts (family members or the police). This alert allows the contacts to quickly identify possible fraud and take appropriate action. If the emotional state is unstable, the server sends an alert with a higher priority, encouraging a quicker response.

[1389] Step 7:

[1390] After an alert is sent, the server logs the results, including the fraudulent activity detected, the alert content, and the destination information, for later analysis and auditing.

[1391] Step 8:

[1392] Users who receive the alert can work with family members or law enforcement to take action to prevent the fraud from progressing, for example by reviewing the content of the call and taking specific steps to rule out the possibility of fraud.

[1393] Through this detailed process flow, users are protected from fraudulent billing scams with high accuracy and can respond quickly and effectively. The addition of an emotion engine enables advanced fraud detection that takes into account the user's mental state, improving overall safety and reliability.

[1394] Example 2

[1395] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1396] In today's world, elderly people and those unfamiliar with technology are increasingly being targeted by fraudulent billing scams. This puts many people at risk of financial loss. However, current security systems have difficulty detecting fraudulent activity in real time and taking appropriate action. Furthermore, because they do not take emotional states into account, they are prone to false positives and limited in the quality of the response. There is a need to solve these problems.

[1397] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1398] In this invention, the server includes: means for allowing a user to perform initial setup and register contact information; means for capturing the user's voice in real time and converting it into text; means for detecting the user's emotional state from the voice data and text data; means for transmitting the converted text data and emotion analysis results to the server; means for analyzing the text using a generative AI model and evaluating the possibility of fraud; and means for sending an alert to pre-registered contacts if the possibility of fraud is detected. This makes it possible to monitor user conversations in real time, detect fraud risks with high accuracy, and respond quickly.

[1399] "Means for users to set up and register contact information" refers to the process by which, after installing the application, users enter contact information for themselves and related parties and store it in a local database for subsequent data transmission and notifications.

[1400] "Means for capturing the user's voice in real time and converting it into text" refers to a function that uses the device's built-in microphone to record the user's speech and converts the voice data into a corresponding text format using voice recognition technology.

[1401] "Means for detecting a user's emotional state from voice data and text data" refers to technology that analyzes the content of a user's speech and voice characteristics to identify the emotions the user is feeling (e.g., anxiety, stress, relaxation).

[1402] "Means for transmitting the converted text data and emotion analysis results to the server" refers to a mechanism for transmitting the text data and emotion state acquired and analyzed on the device to the server using a secure communication protocol.

[1403] "Using a generative AI model to analyze the text and assess the likelihood of fraud" refers to the process of inputting received text data into an AI model based on a neural network or other advanced algorithm to assess the risk of fraud.

[1404] "Means of sending alerts to pre-registered contacts if possible fraudulent activity is detected" refers to a function that sends notifications to pre-registered emergency contacts (e.g., family members, police) if the evaluation results indicate a high possibility of fraudulent activity.

[1405] MODE FOR CARRYING OUT THE INVENTION

[1406] This invention is a system that constantly monitors users' conversations and emotions to detect fraudulent billing scams in real time, and sends alerts to family members and the police if there is a possibility of fraud. This system uses a generative AI model and an emotion engine to predict and detect fraud damage with high accuracy.

[1407] Installing and setting up the dedicated app

[1408] The user first installs the dedicated app and performs the initial setup, which includes registering the user's own contact information as well as emergency contact information for family members, police, etc. At this stage, the registered contact information is securely stored in the device's local database. The information entered by the user will be used for subsequent data transmission and alert generation.

[1409] Voice capture and text transcription

[1410] The device is equipped with a microphone for voice capture, which records the user's conversation in real time. This voice data is converted into text using voice recognition technology. A specific technology that can be used is Google Cloud Speech-to-Text. For example, if a user says, "Tell me the PIN number for my cash card," this voice is converted into text data: "Tell me the PIN number for my cash card."

[1411] emotion recognition

[1412] The converted voice data is analyzed by the device's emotion engine. A specific example of an emotion engine is IBM Watson's Natural Language Understanding, which can identify the user's emotional state (e.g., anxiety, stress, relaxation, etc.) from the content of the user's speech and voice characteristics. For example, the text and voice of a statement such as "Tell me my cash card PIN number" are analyzed and the user is evaluated as feeling anxious.

[1413] Sending data to the server

[1414] The sentiment analysis results and text data are sent to the server using a secure communication protocol such as HTTPS. The data sent includes the converted text data and the emotional state, allowing even sensitive information to be handled safely.

[1415] Potential fraud analysis

[1416] The server feeds the received data into a generative AI model (e.g., GPT-4) to assess the likelihood of fraud, for example using the following prompt:

[1417] "Analyze the following conversation and assess the likelihood of fraud. The user's emotion is 'anxiety'. Conversation: 'Please tell me your cash card PIN number.'"

[1418] The analysis results obtained in this way are compared with a database of past fraud cases to detect signs of fraud.

[1419] Generate and send alerts

[1420] If the generative AI model's analysis determines that fraud is likely, the server generates an alert. This alert includes information about the potential fraud, the content of the conversation, and the user's emotional state. The alert is sent to pre-registered contacts (e.g., family, police). The alert notification may include a message such as:

[1421] "A potentially fraudulent conversation has been detected. The user is concerned. Please check immediately."

[1422] This system monitors user conversations in real time, detects fraud risks with high accuracy, and responds quickly. It can provide effective crime prevention measures, especially for individuals at high risk of becoming a victim of fraud, such as the elderly. By using an emotion engine, highly accurate fraud detection is achieved that takes into account the user's mental state, reducing false positives and improving the quality of responses.

[1423] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1424] Step 1: Install and set up the dedicated app

[1425] The user installs the app. After installation, the app displays an initial setup screen where the user enters contact information for themselves, their family, the police, etc. The entered contact information is saved in the device's local database. The entered data is used in subsequent processing of the app.

[1426] Input: User's contact information

[1427] Output: Contact information stored in a local database

[1428] Specific behavior:

[1429] The user installs the app and accesses the initial setup screen.

[1430] You will be prompted to enter contact information for your family members.

[1431] When the user enters contact information and presses the "Save" button, the contact information is saved to a local database.

[1432] Step 2: Capture audio and convert it to text

[1433] The device uses a built-in microphone to record the user's speech in real time, and this voice data is converted to text using the Google Cloud Speech-to-Text API.

[1434] Input: User's voice data

[1435] Output: Text data

[1436] Specific behavior:

[1437] When the user starts speaking, the device's microphone starts capturing it.

[1438] For example, voice data saying "Tell me the PIN number for my cash card" is acquired.

[1439] The voice data is passed to the Google Cloud Speech-to-Text API and converted into text: "What is my cash card PIN?"

[1440] Step 3: Emotion Recognition

[1441] The device passes the converted text data to an emotion engine that uses IBM Watson's Natural Language Understanding to detect the user's emotional state from the voice and text data.

[1442] Input: Text data and audio data

[1443] Output: User's emotional state

[1444] Specific behavior:

[1445] The text data "Please tell me the PIN number for my cash card" is input to the emotion engine.

[1446] The emotion engine analyzes the tone and content of the user's voice.

[1447] As a result of the analysis, it is detected that the user is feeling "anxiety."

[1448] Step 4: Sending data to the server

[1449] The device sends the converted text data and the emotion analysis results to the server using the HTTP protocol.

[1450] Input: Text data and sentiment analysis results

[1451] Output: Data sent to the server

[1452] Specific behavior:

[1453] The text data generated on the device, "Please tell me your cash card PIN number" and the emotional state "anxiety," are structured in JSON format.

[1454] This data is sent to the server using the HTTPS protocol.

[1455] Step 5: Fraud Potential Analysis

[1456] The server inputs the received data into a generative AI model (e.g., GPT-4), which compares it with a database of past fraud cases to assess the likelihood of fraud.

[1457] Input: Text data, sentiment analysis results

[1458] Output: Fraud likelihood assessment result

[1459] Specific behavior:

[1460] The server generates a prompt. For example, "Analyze the following conversation and assess the likelihood of fraud. The user's emotion is detected as 'anxiety.' Conversation: 'Please tell me your cash card PIN number.'"

[1461] Text data along with a prompt sentence is input into the generative AI model.

[1462] The generative AI model performs the analysis and returns a result that indicates there is a high probability of fraud.

[1463] Step 6: Generate and send an alert

[1464] If the server determines that fraud is likely, it generates an alert. This alert includes information about the potential fraud, the content of the conversation, and the user's emotional state. The alert is sent to pre-registered contacts (family members or the police).

[1465] Input: Fraud possibility assessment results, registered contact information

[1466] Output: Alert notification sent

[1467] Specific behavior:

[1468] The server generates an alert message saying, "A potentially fraudulent conversation has been detected. The user is concerned. Please check immediately."

[1469] Alerts will be sent via email and SMS based on the registered contact information.

[1470] This system makes it possible to monitor user conversations in real time, detect fraud risks with high accuracy, and respond quickly.

[1471] (Application example 2)

[1472] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1473] Elderly people and those living alone are at increased risk of falling victim to fraudulent billing scams. These fraudulent activities are becoming more sophisticated, making simple fraud detection systems insufficient. Furthermore, more accurate judgments are required by taking into account the user's emotional state, but current systems are lacking in this area. Therefore, the object of this invention is to provide a security system that constantly monitors the user's conversations and emotions, detects fraudulent activities in real time with high accuracy, and responds quickly and appropriately.

[1474] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for capturing a user's voice in real time and converting it into text, means for analyzing the text using a generative AI model and evaluating the possibility of fraud, means for sending an alert to pre-registered contacts when the possibility of fraud is detected, means for evaluating the user's emotional state using an emotion analysis engine, and means for including the user's emotional state in the alert along with the possibility of fraud. This enables highly accurate fraud detection and rapid response that takes the user's mental state into consideration.

[1475] "Means for capturing and converting user speech into text in real time" refers to a combination of devices and algorithms for quickly capturing speech emitted by a user and converting it into text information.

[1476] "Means for analyzing said text using a generative AI model to assess the likelihood of fraudulent activity" refers to a system that uses AI technology to analyze text data and determine whether it is related to fraudulent activity.

[1477] "A means of sending alerts to pre-registered contacts when possible fraudulent activity is detected" is a function that sends an alert to pre-set contacts such as family members or the police when signs of fraud are detected.

[1478] "Means for assessing a user's emotional state using an emotion analysis engine" refers to technology that analyzes voice and text data to recognize and evaluate the emotions felt by the user.

[1479] The "means for including the user's emotional state along with the possibility of fraud in the alert" is a mechanism for including the user's emotional state in a warning message indicating the possibility of fraud.

[1480] This invention is a system that monitors user conversations and emotions in real time, detects fraudulent billing with high accuracy, and promptly sends an alert. Specific embodiments for implementing this invention will be described below.

[1481] 1. System Configuration

[1482] Audio capture device:

[1483] Devices such as smartphones have built-in microphones that are used to capture the user's voice in real time.

[1484] Speech-to-text methods:

[1485] The captured audio data is converted to text using the Google Cloud Speech-to-Text API, which makes the audio information available as text.

[1486] Analysis methods using generative AI models:

[1487] The converted text data is then analyzed using a generative AI model, such as OpenAI's GPT-3, which compares it with a database of past fraud cases to assess the likelihood of fraud.

[1488] Sentiment Analysis Engine:

[1489] It uses IBM Watson's emotion analysis engine to analyze the user's emotional state, for example, assessing the user's stress, anxiety, or level of relaxation from voice data.

[1490] Alerts sent via:

[1491] If a fraudulent activity is assessed as likely, an alert will be sent to pre-registered contacts such as family members or the police, and the alert will include information indicating the possibility of fraud and the user's emotional state.

[1492] 2. Specific examples of processing

[1493] Capturing user conversations and emotions

[1494] Suppose a user says into their smartphone, "What is my PIN number for my cash card?" The smartphone's microphone captures this speech and sends it in real time to the Google Cloud Speech-to-Text API, where it is converted into text.

[1495] Emotion Analysis

[1496] The converted text and voice data is sent to IBM Watson's sentiment analysis engine to assess whether the user is feeling anxious.

[1497] Fraud Analysis

[1498] A generative AI model (OpenAI GPT-3) analyzes the converted text data and compares it with a database of past fraud cases. For example, the following prompt sentence is used:

[1499] Example prompt sentence:

[1500] Please rate the likelihood that the following conversation text is a scam.

[1501] Text: What is your cash card PIN number?

[1502] If the generative AI model's analysis determines that there is a high possibility of fraud, the system will generate an alert.

[1503] Sending alerts

[1504] The alert will be sent to pre-registered family members and the police. The alert content will include information about the possibility of fraud and the user's concerns. For example, an alert may be sent with the content, "A potentially fraudulent conversation has been detected. The user is concerned. Please check immediately."

[1505] The system of the present invention can protect elderly people and those living alone from sophisticated fraud. Furthermore, by taking into account the user's emotional state, it is expected to achieve more accurate fraud detection and reduce false positives.

[1506] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1507] Step 1:

[1508] The user speaks into the smartphone. The smartphone's microphone captures the voice in real time. The input is voice data, and the output is the captured voice. For example, the user might record a voice saying, "Tell me my cash card PIN."

[1509] Step 2:

[1510] The smartphone sends the captured voice data to the Google Cloud Speech-to-Text API, which converts it into text data in real time. The input is the captured voice data, and the output is the text data corresponding to that voice. Specifically, the voice saying "Tell me my cash card PIN" is converted into text "Tell me my cash card PIN."

[1511] Step 3:

[1512] The converted text data and voice data are sent to IBM Watson's emotion analysis engine. The emotion analysis engine analyzes this data and evaluates the user's emotional state. The input is text data and voice data, and the output is analyzed emotional state data. In concrete terms, the text "Tell me my cash card PIN" is evaluated as indicating that the user is feeling anxious.

[1513] Step 4:

[1514] A generative AI model (OpenAI GPT-3) analyzes the converted text data. The input is the text data and the sentiment analysis results, and the output is an assessment of the likelihood of fraud. The prompt used is "Please assess the likelihood that the following conversational text is fraudulent. Text: Tell me your cash card PIN." The analysis results are obtained.

[1515] Step 5:

[1516] If the server determines that there is a high possibility of fraud, it generates an alert. The input is the result of the fraud possibility assessment and the user's emotional state data, and the output is the alert content. The alert includes information about the possibility of fraud and the user's anxiety. Specifically, the alert content is "A potentially fraudulent conversation has been detected. The user is feeling anxious. Please check immediately."

[1517] Step 6:

[1518] The server sends the alert content to pre-registered contacts such as family members and the police. The input is the alert content, and the output is the sent alert. Specifically, alerts are sent via email or messaging apps.

[1519] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1520] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1521] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1522] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1523] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1524] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1525] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1526] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1527] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1528] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1529] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1530] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1531] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1532] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1533] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1534] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1535] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1536] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1537] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1538] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1539] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1540] The following is further disclosed regarding the above embodiment.

[1541] (Claim 1)

[1542] a means for capturing and converting a user's speech into text in real time;

[1543] means for analyzing the text using a generative AI model to assess likelihood of fraud;

[1544] A means of sending alerts to pre-registered contacts when potential fraud is detected; and

[1545] A system including:

[1546] (Claim 2)

[1547] 10. The system of claim 1, wherein the alert is sent to both family members and police.

[1548] (Claim 3)

[1549] The system of claim 1, wherein the generative AI model that assesses the likelihood of fraudulent activity performs analysis by comparing it with a database of past fraud cases.

[1550] "Example 1"

[1551] (Claim 1)

[1552] a means for capturing and converting a user's speech into text in real time;

[1553] means for analyzing the text using a generative AI model to assess likelihood of fraud;

[1554] A means of sending alerts to pre-registered contacts when potential fraud is detected; and

[1555] means for matching said text with a database of past fraud cases;

[1556] means for transmitting said text to a server using encrypted communications;

[1557] A system including:

[1558] (Claim 2)

[1559] 10. The system of claim 1, wherein the alert is sent to both family members and police.

[1560] (Claim 3)

[1561] The system of claim 1, wherein the generative AI model that assesses the likelihood of fraudulent activity performs analysis by comparing it with a database of past fraud cases.

[1562] "Application Example 1"

[1563] (Claim 1)

[1564] a means for capturing and converting a user's speech into text in real time;

[1565] means for analyzing the text using a generative AI model to assess likelihood of fraud;

[1566] A means of sending alerts to pre-registered contacts when potential fraud is detected; and

[1567] It is installed on smartphones and is a means of monitoring users' calls in real time.

[1568] A means of recording user conversations and analyzing the content using a generative AI model;

[1569] A means of assessing indicators of fraud by checking a database of past fraud cases;

[1570] A means of communication to send alerts immediately;

[1571] A system including:

[1572] (Claim 2)

[1573] 10. The system of claim 1, wherein the alert is sent to both family members and police.

[1574] (Claim 3)

[1575] The system of claim 1, wherein the generative AI model that assesses the likelihood of fraudulent activity performs analysis by comparing it with a database of past fraud cases.

[1576] "Example 2: Combining Emotion Engines"

[1577] (Claim 1)

[1578] a means for a user to configure initial settings and register contact information;

[1579] a means for capturing and converting a user's speech into text in real time;

[1580] means for detecting a user's emotional state from the voice data and text data;

[1581] means for transmitting the converted text data and the emotion analysis result to a server;

[1582] means for analyzing the text using a generative AI model to assess likelihood of fraud;

[1583] A means of sending alerts to pre-registered contacts when potential fraud is detected; and

[1584] A system including:

[1585] (Claim 2)

[1586] 10. The system of claim 1, wherein the alert is sent to both family members and police.

[1587] (Claim 3)

[1588] The system of claim 1, wherein the generative AI model that assesses the likelihood of fraudulent activity performs analysis by comparing it with a database of past fraud cases.

[1589] "Application example 2 when combining emotion engines"

[1590] (Claim 1)

[1591] a means for capturing and converting a user's speech into text in real time;

[1592] means for analyzing the text using a generative AI model to assess likelihood of fraud;

[1593] A means of sending alerts to pre-registered contacts when potential fraud is detected; and

[1594] means for assessing the emotional state of a user using an emotion analysis engine;

[1595] means for including in the alert the user's emotional state along with the likelihood of fraud;

[1596] A system including:

[1597] (Claim 2)

[1598] 10. The system of claim 1, wherein the alert is sent to both family members and police.

[1599] (Claim 3)

[1600] The system of claim 1, wherein the generative AI model that assesses the likelihood of fraudulent activity performs analysis by comparing it with a database of past fraud cases. [Explanation of symbols]

[1601] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a means for capturing and converting a user's speech into text in real time; means for analyzing the text using a generative AI model to assess likelihood of fraud; A means of sending alerts to pre-registered contacts when potential fraud is detected; and A system including:

2. The system of claim 1 , wherein the alert is sent to both family members and police.

3. The system of claim 1, wherein the generative AI model that evaluates the likelihood of fraudulent activity performs analysis by comparing it with a database of past fraud cases.

Images