System

Abstract

To provide a system.SOLUTION: A system includes: means for confirming contents of an email or a phone call; generative AI for inferring whether the confirmed contents are specialized fraud or not based on a prompt sentence instructing it to infer whether the confirmed contents are specialized fraud or not; means for generating advice to a person in question and his / her family including the possibility of fraud and countermeasures based on the inference by the generative AI; and means for notifying the person and his / her family of the generated advice.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Special frauds occur when an unusual situation occurs, which impairs the victim's judgment and prevents them from consulting with others, allowing them to quickly commit fraud. There is a need for methods to prevent such special frauds. [Means for solving the problem]

[0005] We provide a system that combines a means to check the content of emails and phone calls, a generative AI that makes an inference if the content is suspected to be special fraud, and a means to advise the individual and their family on the possibility of fraud and countermeasures based on that inference.This system enables the early detection and prevention of special fraud. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] In one embodiment of the present invention, software that analyzes communication data is used as a means for checking the contents of emails and phone calls. This software converts the text of emails and voice data from phone calls into text and analyzes the contents.

[0029] "Example 2"

[0030] A machine learning model is used as a generative AI to determine if the content is suspected to be special fraud. This model learns typical patterns of special fraud and determines whether the input text matches those patterns. For example, if the text contains phrases such as "Please transfer money" or "A family member of yours has been in an accident," it is determined to be highly likely to be special fraud.

[0031] "Example 3"

[0032] Based on this estimation, a notification system is used to advise the individual and their family of the possibility of fraud and countermeasures. If the system determines that there is a possibility of special fraud, it sends a notification to the individual and their registered family. The notification includes information about the possibility of fraud and specific guidelines for protecting oneself from fraud. For example, advice such as "You have received a suspicious email. Please avoid providing personal information or transferring money" is provided.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: Software that analyzes communication data converts the text of emails and audio data from phone calls into text.

[0036] Step 2: Analyze the text to see if it contains typical patterns of special fraud.

[0037] "Example 2"

[0038] Step 1: The machine learning model learns typical patterns of specialized fraud.

[0039] Step 2: Use the trained model to determine whether the input text matches a special fraud pattern.

[0040] "Example 3"

[0041] Step 1: If it is determined that there is a possibility of special fraud, the notification system will send a notification to the individual and their registered family members.

[0042] Step 2: The notice will include information about the potential fraud and specific actions you can take to protect yourself from it.

[0043] Example 1

[0044] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0045] In modern society, communication via email and telephone is on the rise, but some of this communication may contain malicious content, such as special fraud. This increases the risk of individuals and families becoming victims of fraud. Traditional methods face the challenge of quickly and accurately detecting these frauds and providing appropriate countermeasures.

[0046] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0047] In this invention, the server includes means for checking the content of emails and telephone calls, means for converting the content into text, means for analyzing the converted text, means for generating an appropriate response or action using a generative AI model based on the analysis results, a generative AI for making an inference if the content is suspected to be special fraud, and means for advising the user and their family on the possibility of fraud and countermeasures based on the inference. This makes it possible to quickly and accurately detect the possibility of special fraud from communications via emails and telephones and provide the user with appropriate countermeasures.

[0048] "Means for checking the contents of emails and phone calls" refers to a device or software that has the function of acquiring the contents of emails and phone calls received by the user and sending them to a server for analysis.

[0049] The "means for converting the content into text" refers to a device or software for converting the text of an acquired email or voice data from a telephone call into text format.

[0050] The "means for analyzing the textual content" refers to a device or software for analyzing the textual data and extracting specific keywords or phrases.

[0051] "Means for generating appropriate responses or actions using a generative AI model" refers to a device or software that uses a generative AI model to generate appropriate responses or actions for a user based on the analysis results.

[0052] "Generative AI that makes the inference when the content is suspected to be special fraud" is a generative AI model that analyzes the textual content and checks whether it contains typical patterns of special fraud.

[0053] "Means for advising the individual and their family of the possibility of fraud and countermeasures based on the presumption" refers to a device or software that notifies the user and their family of the risk of fraud and specific countermeasures when there is a possibility of special fraud.

[0054] MODE FOR CARRYING OUT THE INVENTION

[0055] The present invention relates to a system for checking the contents of emails and telephone calls to detect the possibility of special fraud. A specific embodiment of this system will be described below.

[0056] System configuration

[0057] The system consists of the following main components:

[0058] 1. Server

[0059] 2. Terminal

[0060] 3. Users

[0061] Hardware and software used

[0062] Server: A server with high-performance computing power is used. The server collects and analyzes data from mail servers and telephone systems.

[0063] Device: A device such as a computer or smartphone that a user uses.

[0064] software:

[0065] Email retrieval software: Retrieves email data using IMAP or POP3 protocols.

[0066] Speech recognition software: Converts voice data into text using software such as the Google® Cloud Speech-to-Text API.

[0067] Natural Language Processing (NLP) software: Use spaCy or NLTK to analyze text data.

[0068] Generative AI models: Use OpenAI (registered trademark) GPT-3 (registered trademark), etc., to generate appropriate responses and actions based on analysis results.

[0069] System Operation

[0070] 1. Data Collection:

[0071] The server retrieves new emails from the user's email server, specifically by using the IMAP or POP3 protocol to collect email data.

[0072] The server retrieves call recording data from the VoIP system.

[0073] 2. Data conversion:

[0074] The server converts the collected email data into text format and extracts the text data from the email body and attachments.

[0075] The server uses voice recognition software to convert the call recording into text.

[0076] 3. Data Analysis:

[0077] The server uses natural language processing (NLP) software to analyze the text data, which analyzes the content of emails and phone calls and extracts specific keywords and phrases.

[0078] The server uses the generative AI model to generate appropriate responses and actions for the user based on the analysis results.

[0079] Specific examples

[0080] Consider the case where a user receives an email about the progress of a new project.

[0081] 1. Data Collection:

[0082] The server retrieves new mail from the user's mail server.

[0083] The server retrieves call recording data from the VoIP system.

[0084] 2. Data conversion:

[0085] The server converts the body of the email into text format.

[0086] The server uses voice recognition software to convert the call recording into text.

[0087] 3. Data Analysis:

[0088] The server analyzes the contents of the converted email and extracts the keyword "project progress."

[0089] The server analyzes the text of the call and extracts relevant information.

[0090] 4. Response Generation:

[0091] The server uses a generative AI model to input the following prompt:

[0092] Prompt: "Analyze an email about the progress of a new project and summarize the key points."

[0093] The generative AI model summarizes the contents of the email and generates a response to the user, such as "The project is progressing smoothly and the next step is ____."

[0094] In this way, the server can analyze the communication data and provide useful information to the user.

[0095] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0096] Step 1: Data collection

[0097] The server collects communication data from the user's mail server and telephone system. Specifically, it retrieves email data using the IMAP or POP3 protocol, and retrieves telephone voice data using the VoIP protocol.

[0098] Input: User's email account information, VoIP system connection information

[0099] Output: Acquired email data and audio data

[0100] Specific behavior:

[0101] The server accesses the user's email account and checks for new email.

[0102] The server connects to the VoIP system and retrieves the call recording data.

[0103] Step 2: Data conversion

[0104] The server converts the collected email data into text format, extracts text data from the email body and attachments, and uses speech recognition software (e.g., Google Cloud Speech-to-Text API) to convert voice data into text.

[0105] Input: Acquired email data and audio data

[0106] Output: Text email data, text-converted audio data

[0107] Specific behavior:

[0108] The server analyzes the body of the email and extracts the text data.

[0109] The server invokes speech recognition software to convert the voice data into text.

[0110] Step 3: Data analysis

[0111] The server uses natural language processing (NLP) software (e.g., spaCy, NLTK) to analyze the text data, which allows it to analyze the content of emails and phone calls and extract specific keywords and phrases.

[0112] Input: Text email data, audio data converted to text

[0113] Output: Extracted keywords and phrases, analysis results

[0114] Specific behavior:

[0115] The server uses NLP software to analyze the text data and extract important keywords.

[0116] The server classifies the contents of emails and phone calls based on the extracted keywords.

[0117] Step 4: Response Generation

[0118] The server uses a generative AI model (e.g., OpenAI GPT-3) to generate appropriate responses and actions for the user based on the analysis results.

[0119] Input: Analysis results, extracted keywords and phrases

[0120] Output: Response or action to the user

[0121] Specific behavior:

[0122] The server inputs the following prompt sentence into the generative AI model:

[0123] Prompt: "Analyze an email about the progress of a new project and summarize the key points."

[0124] The generative AI model summarizes the contents of the email and generates a response to the user, such as "The project is progressing smoothly and the next step is ____."

[0125] In this way, the server can analyze the communication data and provide useful information to the user.

[0126] (Application example 1)

[0127] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0128] In modern society, special frauds via email and telephone are on the rise, causing significant damage to individuals and businesses. In particular, frauds using sophisticated methods are difficult to detect using conventional methods, making it difficult to prevent damage before they occur. Furthermore, even if a possible fraud is detected, there is a lack of means to quickly provide specific countermeasures. This can prevent victims from taking appropriate measures, and the damage can become more severe. Therefore, there is a need for a system that can analyze the content of emails and telephone calls, detect possible fraud with high accuracy, and quickly provide countermeasures.

[0129] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0130] In this invention, the server includes a means for checking the content of emails and phone calls, a generative AI that makes a prediction if the content is suspected to be special fraud, a means for advising the individual and their family on the possibility of fraud and countermeasures based on the prediction, a means for converting voice data into text, a means for converting email data into text, and a means for analyzing the text data. This makes it possible to analyze the content of emails and phone calls in real time, detect the possibility of fraud with high accuracy, and quickly provide specific countermeasures.

[0131] "Means for checking the contents of emails and phone calls" refers to devices or software for analyzing the contents of received emails and phone calls and obtaining that information.

[0132] "Generative AI that makes the inference if the content is suspected to be special fraud" is artificial intelligence that analyzes the content of received emails and phone calls and determines whether the content constitutes special fraud.

[0133] "Means for advising the individual and their family of the possibility of fraud and countermeasures based on the said presumption" refers to a device or software that notifies the user and their family of the risk of fraud and countermeasures when generative AI determines that there is a possibility of special fraud.

[0134] "Means for converting voice data into text" refers to a device or software for converting voice data, such as telephone data, into text information.

[0135] "Means for converting email data into text" refers to a device or software that analyzes the content of received emails and obtains them as text information.

[0136] "Means for analyzing text data" refers to a device or software for analyzing text data and evaluating its content.

[0137] A system for implementing this invention has the following configuration: The server includes means for checking the content of emails and phone calls, a generative AI that makes an inference if the content is suspected to be special fraud, means for advising the individual and their family on the possibility of fraud and countermeasures based on the inference, means for converting voice data into text, means for converting email data into text, and means for analyzing the text data.

[0138] Hardware and Software Configuration

[0139] Hardware: Smartphone (iOS or ANDROID (registered trademark))

[0140] software:

[0141] Python 3.x

[0142] speech_recognition library: Used to convert speech data into text

[0143] email library: Used to parse email data

[0144] The transformers library: used to parse text using generative AI models

[0145] Data processing and calculation

[0146] 1. Converting audio data to text:

[0147] The server uses the speech_recognition library to convert voice data, such as telephone calls, into text information, which is then used to obtain the voice data as text data.

[0148] 2. Converting email data to text:

[0149] The server uses the email library to analyze the content of the received email and obtains it as text information. This allows the email data to be obtained as text data.

[0150] 3. Text data analysis:

[0151] The server uses the transformers library to analyze the text data, evaluate its content, and use a generative AI model to determine whether the content constitutes a fraudulent activity.

[0152] 4. Possible fraud and advice on how to deal with it:

[0153] If the generative AI model determines that there is a possibility of fraud, the server notifies the user and their family of the risk of fraud and how to prevent it, providing specific guidelines for action so that the user can take appropriate action.

[0154] Specific examples

[0155] Corporate security personnel analyze the content of received emails and phone calls in real time to detect phishing scams and spam.

[0156] When individual users receive suspicious emails or phone calls, they can use this system to analyze the content and avoid security risks.

[0157] Prompt Sentence Examples

[0158] "Please analyze the contents of this email and let me know if it is a possible phishing scam."

[0159] "Please transcribe this call and let me know if it's spam."

[0160] In this way, it is possible to analyze the contents of emails and phone calls in real time, detect possible fraud with high accuracy, and quickly provide specific countermeasures.

[0161] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0162] Step 1:

[0163] The server receives email data or telephone voice data received from the user. The input is email data or voice data, and the output is the data as is. Specifically, the server sends the voice file of the email or telephone call received by the user to the server.

[0164] Step 2:

[0165] The server converts the telephone voice data into text information using a means of converting voice data to text. The input is voice data and the output is text data. Specifically, the server uses the speech_recognition library to analyze the voice data and obtain it as text information.

[0166] Step 3:

[0167] The server uses a means to convert email data into text and obtains the contents of the received email as text information. The input is email data and the output is text data. Specifically, the email library is used to parse the body of the email and obtain it as text information.

[0168] Step 4:

[0169] The server analyzes the textualized data using a means for analyzing text data. The input is text data, and the output is the analysis result. Specifically, the server uses the transformers library to input text data into a generative AI model and evaluates its content.

[0170] Step 5:

[0171] If the generative AI model determines that there is a possibility of fraud, the server notifies the user and their family of the risk of fraud and how to prevent it. The input is the analysis result, and the output is a notification message. Specifically, the server notifies the user that there is a possibility of fraud and provides specific guidelines for action.

[0172] Step 6:

[0173] The user receives a notification from the server and takes specific measures to address the risk of fraud. The input is the notification message, and the output is the user's actions. Specifically, by taking appropriate action in accordance with the notification, the user can prevent fraud before it occurs.

[0174] Example 2

[0175] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0176] In recent years, special fraud methods have become more sophisticated, and many people have fallen victim to them. Elderly people and those unfamiliar with the Internet are particularly susceptible to fraud. In order to prevent such fraud damage, a system is needed that can quickly and accurately detect possible fraud and provide appropriate countermeasures. However, current systems have low fraud detection accuracy and provide insufficient countermeasures, making it difficult to effectively prevent fraud.

[0177] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0178] In this invention, the server includes a means for a user to input text data, a means for a terminal to send the text data to the server, a means for the server to receive the text data and input it to a generative AI model, a means for the generative AI model to analyze the text data and generate a result, a means for the server to send the analysis result to the terminal, a means for the terminal to display the analysis result to the user, a generative AI that makes an inference if the content is suspected to be special fraud, and a means for advising the person and their family of the possibility of fraud and countermeasures based on the inference. This makes it possible to quickly and accurately detect the possibility of fraud and provide the user with appropriate countermeasures.

[0179] A "user" is an individual or organization that uses the system to input text data and receive analysis results.

[0180] A "terminal" is a device that allows a user to input text data, send it to a server, and display the analysis results. Specifically, this applies to smartphones and personal computers.

[0181] A "server" is a computer system that receives text data sent from a terminal, analyzes it using a generative AI model, and returns the results to the terminal.

[0182] A "generative AI model" is an artificial intelligence model that uses machine learning algorithms to learn typical patterns of special fraud and determine whether input text data matches those patterns.

[0183] "Text data" is character string information entered by the user, and is the message or sentence to be analyzed.

[0184] The "analysis result" is the judgment result of whether the text data constitutes special fraud, which is generated after the generative AI model analyzes the text data.

[0185] "Special fraud" is a type of fraud in which specific patterns or phrases are used to deceive victims and illegally obtain money or information.

[0186] "Advice tools" is a function that notifies users and their families of possible fraud based on the analysis results and provides specific guidelines for protecting themselves from fraud.

[0187] This invention is a system for detecting special frauds and providing countermeasures. The system includes a series of processes in which a user inputs text data, a terminal sends the data to a server, the server analyzes the data using a generative AI model, returns the results to the terminal, and the terminal displays the analysis results to the user.

[0188] Hardware and software used

[0189] Hardware: Servers (e.g., virtual machines for cloud computing services), devices (e.g., smartphones, personal computers)

[0190] Software: Generative AI models (e.g., GPT-3), communication protocols (e.g., HTTP / HTTPS)

[0191] Specific operation of the system

[0192] User Actions

[0193] A user inputs text data into an input field of a terminal. For example, the user inputs a message such as "Please transfer money." This message is a target for detecting the possibility of special fraud.

[0194] Device behavior

[0195] When the user presses the send button, the terminal sends the entered text data to the server. HTTP / HTTPS is used as the communication protocol. Specifically, the terminal sends a POST request to the server and includes the text data in the request body.

[0196] Server Operation

[0197] The server receives the text data sent from the device. The received text data is input into a generative AI model. For example, GPT-3 is used as the generative AI model. This model has learned typical patterns of special fraud.

[0198] How generative AI models work

[0199] The generative AI model analyzes the input text data and determines whether its content matches the pattern of specialized fraud. For example, if the message contains the phrase "Please transfer money," the model determines that the message is likely to be specialized fraud.

[0200] Server Operation

[0201] When the generative AI model generates an analysis result, the server sends the result to the terminal. Specifically, the server includes the analysis result in the response body and sends it to the terminal as an HTTP / HTTPS response.

[0202] Device behavior

[0203] The terminal displays the analysis result received from the server to the user. For example, if the analysis result indicates that "this message may be a special fraud," the terminal displays that message as a warning to the user.

[0204] Specific examples

[0205] Prompt Sentence Examples

[0206] "Please judge whether the following message is a special type of fraud: 'Please transfer money.'"

[0207] When this prompt is input into a generative AI model, the model determines whether the message matches the typical pattern of a special fraud, and returns a response such as, "This message may be a special fraud."

[0208] In this way, the system can quickly and accurately detect the possibility of special fraud and provide the user with appropriate countermeasures.

[0209] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0210] Step 1:

[0211] The user inputs text data.

[0212] A user inputs text data into an input field of a terminal. For example, the user inputs a message such as "Please transfer money." This input data is used to detect the possibility of special fraud.

[0213] Step 2:

[0214] The terminal transmits the text data to the server.

[0215] When the user presses the send button, the terminal sends the entered text data to the server. HTTP / HTTPS is used as the communication protocol. Specifically, the terminal sends a POST request to the server and includes the text data in the request body. The input is the text data entered by the user, and the output is the request sent to the server.

[0216] Step 3:

[0217] The server receives the text data and inputs it into the generative AI model.

[0218] The server receives text data sent from the device. The received text data is input into a generative AI model. For example, GPT-3 is used as the generative AI model. This model has learned typical patterns of special fraud. The input is the text data received from the device, and the output is the data input into the generative AI model.

[0219] Step 4:

[0220] A generative AI model analyzes the text data and generates results.

[0221] The generative AI model analyzes input text data and determines whether its content matches the pattern of special fraud. For example, if the message contains the phrase "Please transfer money," the model determines that it is likely to be a special fraud. The input is text data entered from the server, and the output is the analysis result.

[0222] Step 5:

[0223] The server sends the analysis results to the device.

[0224] Once the generative AI model generates the analysis results, the server sends them to the terminal. Specifically, the server includes the analysis results in the response body and sends them to the terminal as an HTTP / HTTPS response. The input is the analysis results from the generative AI model, and the output is the response sent to the terminal.

[0225] Step 6:

[0226] The terminal displays the analysis results to the user.

[0227] The terminal displays the analysis results received from the server to the user. For example, if the analysis result is "This message may be a special fraud," the terminal displays that message as a warning to the user. The input is the analysis result received from the server, and the output is the warning message displayed to the user.

[0228] (Application example 2)

[0229] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0230] In recent years, the methods of special fraud have become more sophisticated, and many people have fallen victim to them. In particular, elderly people and people with low digital literacy often fall victim to these frauds without realizing it. In order to prevent such frauds from occurring, a system that can detect potential frauds in real time and issue warnings is needed.

[0231] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for checking the content of emails and phone calls, generative AI means for making an inference if the content is inferred to be special fraud, means for advising the person and their family on the possibility of fraud and countermeasures based on the inference, and means for analyzing messages received by the user in real time and issuing a warning if there is a possibility of special fraud. This makes it possible to quickly detect the possibility of special fraud and issue a warning to the user.

[0232] "Means for checking the contents of emails and phone calls" refers to a function that allows the user to analyze the contents of emails and phone calls received and obtain those contents.

[0233] "Generative AI that makes a prediction if the content is suspected to be special fraud" is artificial intelligence that uses a machine learning model that has learned typical patterns of special fraud to determine whether the input text corresponds to special fraud.

[0234] "Means of advising the individual and their family of the possibility of fraud and countermeasures based on the estimation" refers to a function that notifies the user and their family of the possibility of fraud and provides specific countermeasures based on the results of the generative AI's estimation of special fraud.

[0235] "Means of analyzing messages received by users in real time and issuing a warning if there is a possibility of special fraud" is a function that instantly analyzes messages received by users and issues a warning if it is determined that there is a high possibility of special fraud.

[0236] The system for implementing this invention has the function of checking the contents of emails and phone calls received by the user and analyzing the possibility of special fraud in real time. The system includes the following main components.

[0237] 1. How to check emails and phone calls:

[0238] This is a function that allows users to obtain the contents of emails and phone calls they receive. This function is provided by applications installed on devices such as smartphones and computers. For example, the contents of SMS messages and emails can be automatically read.

[0239] 2. Generative AI:

[0240] This is an AI that uses a machine learning model that has learned typical patterns of special fraud to determine whether the input text corresponds to a special fraud. This generative AI uses a model built using the transformers library (e.g., distilbert-base-uncased-finetuned-sst-2-english).

[0241] 3. Advice on potential fraud and how to deal with it:

[0242] This function notifies users and their families of possible fraud based on the results of generative AI's predictions of special frauds and provides specific countermeasures. Notifications are sent via smartphone push notifications and emails.

[0243] 4. Real-time analysis and alerting measures:

[0244] This function instantly analyzes messages received by users and issues a warning if it is determined to be a special fraud. This function works the moment a user receives a message, and immediately displays a warning if there is a possibility of fraud.

[0245] Hardware and software used

[0246] Hardware: Smartphones, PCs

[0247] Software: Python, transformers library, regular expressions (re) module

[0248] Data processing and calculation

[0249] Data processing: The text entered by the user is analyzed using regular expressions to check whether it contains phrases typical of special fraud.

[0250] Data Computing: Using generative AI models to perform sentiment analysis on text to determine likelihood of fraud.

[0251] Specific examples

[0252] For example, if a user receives a message saying, "Your family member has been in an accident. Please transfer money as soon as possible," the system works as follows:

[0253] 1. Get the message content.

[0254] 2. Generative AI analyzes the message and determines whether it is a special fraud.

[0255] 3. If a fraudulent activity is deemed likely, the user will be warned.

[0256] 4. Notify users and their families of possible fraud and specific steps to take.

[0257] Prompt Sentence Examples

[0258] "Your relative has been in an accident. Please transfer the money as soon as possible."

[0259] By inputting this prompt sentence into the model, it is possible to detect possible fraud with a high probability.

[0260] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0261] Step 1:

[0262] The terminal receives messages received by the user. The input is emails or SMS messages received by the user. The terminal receives these messages in text format and passes them to the next processing step.

[0263] Step 2:

[0264] The terminal analyzes the acquired message using regular expressions. The input is the text message acquired in step 1. The terminal checks whether it contains typical phrases of special fraud, and if any such phrases are found, passes the information to the next step. The output is a flag indicating whether or not there is a possibility of fraud.

[0265] Step 3:

[0266] The server analyzes the message using a generative AI model. The input is the text message and fraud likelihood flag obtained in step 2. The server uses a generative AI model (e.g., distilbert-base-uncased-finetuned-sst-2-english) to perform sentiment analysis of the message and determine whether it is likely to be fraudulent. The output is a determination of whether it is likely to be fraudulent.

[0267] Step 4:

[0268] If the server determines that there is a high possibility of fraud, it issues a warning to the terminal. The input is the fraud possibility determination result obtained in step 3. If the server determines that there is a high possibility of fraud, it sends a warning message to the terminal. The output is a warning message that is displayed on the user's terminal.

[0269] Step 5:

[0270] The terminal notifies the user and their family of the possibility of fraud and provides specific countermeasures. The input is the warning message sent from the server in step 4. The terminal notifies the user and their family of the possibility of fraud and provides specific countermeasures. The output is the notification message displayed to the user and their family.

[0271] In this way, a system is realized that analyzes messages received by users in real time and issues a warning if there is a possibility of special fraud.

[0272] Example 3

[0273] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0274] In recent years, the methods of special fraud have become more sophisticated, and there has been an increase in cases where personal information and money are stolen via email and telephone. In order to protect individuals and their families from such fraudulent activities, a system is needed that can quickly and accurately detect possible fraud and provide appropriate countermeasures. However, current systems have low fraud detection accuracy and do not provide sufficient countermeasures, making it difficult to prevent fraud damage before it occurs.

[0275] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0276] In this invention, the server includes means for checking the content of emails and phone calls, a generative AI that makes a prediction if the content is suspected to be special fraud, means for advising the individual and their family of the possibility of fraud and countermeasures based on the prediction, means for preprocessing the content, means for inputting the preprocessed data into the generative AI, means for the generative AI to output the possibility of fraud as a score, means for generating a notification message based on the score, and means for sending the notification message to the individual and their family's devices. This makes it possible to detect the possibility of fraud with high accuracy and quickly provide appropriate countermeasures.

[0277] "Means for checking the contents of emails and phone calls" refers to devices or software that acquire and analyze the contents of emails and phone calls received by the user.

[0278] "Generative AI" is an artificial intelligence model that analyzes collected data and estimates the likelihood of fraud.

[0279] "Means for advising on the possibility of fraud and countermeasures" refers to a device or software that notifies users and their families of the risk of fraud and countermeasures based on the estimation results of the generative AI.

[0280] A "preprocessing means" is a device or software that converts collected data into a format that is easy to analyze.

[0281] "Means for inputting to generative AI" refers to a device or software for passing preprocessed data to generative AI.

[0282] A "means for outputting the likelihood of fraud as a score" is a device or software that allows generative AI to quantify the analysis results and provide the risk of fraud as a score.

[0283] A "means for generating a notification message" is a device or software for generating a warning message for the user and their family based on the fraud risk score.

[0284] The "means for transmitting a notification message to the user's and family's terminals" refers to a device or software for transmitting the generated notification message to the user's and family's devices.

[0285] This invention is a system that detects the risk of special fraud and provides appropriate countermeasures to users and their families. This system consists of multiple components, including a server, a terminal, and a generative AI model.

[0286] Hardware and software used

[0287] Server: A central server for data processing and notification sending, specifically a virtual server from a cloud service provider.

[0288] Devices: Devices on which you and your family members receive notifications, such as smartphones and tablets.

[0289] Generative AI model: An AI model for estimating the likelihood of fraud. Specifically, a generative AI model based on natural language processing is used.

[0290] Data processing and calculation

[0291] 1. Data collection: The server collects the contents of the user's emails and messages. This includes data collection with the user's consent. Specifically, it retrieves emails from the mail server using the IMAP protocol.

[0292] 2. Data preprocessing: The server preprocesses the collected data. Specifically, it cleans the text (removes unnecessary spaces and special characters). For preprocessing, a natural language processing library is used.

[0293] 3. Fraud probability estimation: The server inputs the preprocessed data into the generative AI model to estimate the likelihood of fraud. The generative AI model outputs a score indicating whether the likelihood of fraud is high.

[0294] 4. Notification Generation: If fraud is determined to be likely, the server generates a notification message stating the possibility of fraud and providing specific instructions for action.

[0295] 5. Notification sending: The server sends the generated notification to the user and registered family members' devices. The notification is sent using a messaging service.

[0296] Specific examples

[0297] User email reads: "Your account has been compromised. Please click this link to verify."

[0298] Prompt for the generative AI model: "Please determine if the following email is likely to be fraudulent. Email content: 'Your account has been compromised. Click this link to verify.'"

[0299] Notification message: "You have received a suspicious email. Please do not provide any personal information or transfer any money."

[0300] This system allows users and their families to receive specific advice on how to protect themselves from the risk of fraud. The server automatically performs a series of processes from collecting data to sending notifications, so users can avoid the risk of fraud without having to perform any special operations. The flow of the identification process in Example 3 will be described with reference to FIG. 15.

[0301] Step 1:

[0302] Data collection

[0303] A user receives an email or message.

[0304] The server retrieves new mail from the user's mail server using the IMAP protocol.

[0305] Input: User's email account information and new incoming email.

[0306] Output: The retrieved email data.

[0307] What happens: The server accesses the user's email account, downloads new emails, and temporarily stores them.

[0308] Step 2:

[0309] Data Preprocessing

[0310] Preprocessing of mail data retrieved by the server.

[0311] Input: The retrieved email data.

[0312] Output: Preprocessed text data.

[0313] What it does: The server removes unnecessary spaces and special characters from the text of the email and tokenizes the text using a natural language processing library.

[0314] Step 3:

[0315] Estimated likelihood of fraud

[0316] The server inputs the preprocessed data into the generative AI model.

[0317] Input: Preprocessed text data.

[0318] Output: Fraud likelihood score.

[0319] How it works: The server inputs the preprocessed data into the generative AI model, which outputs a fraud probability score. The generative AI model uses natural language processing techniques to analyze the text and assess the risk of fraud.

[0320] Step 4:

[0321] Notification generation

[0322] The server generates a notification message based on the fraud likelihood score.

[0323] Input: Fraud likelihood score.

[0324] Output: Informational message.

[0325] Specific Actions: If the fraud likelihood score exceeds a certain threshold, the server generates a warning message for the user and their family members, including the fraud risk and specific instructions for action.

[0326] Step 5:

[0327] Send notifications

[0328] The server generates a notification message and sends it to the terminals of the user and registered family members.

[0329] Input: Notification message.

[0330] Output: Notifications displayed on the user's and family's devices.

[0331] Specific operation: The server uses a messaging service to send the generated notification message to the user's and family's smartphones and tablets. The notification is displayed on the device so that the user and family can check it.

[0332] In this way, users and their families can receive specific advice on how to protect themselves from the risk of fraud.

[0333] (Application example 3)

[0334] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0335] In recent years, the methods used by special fraudsters have become more sophisticated, with many people falling victim to them. Elderly people and those unfamiliar with the Internet are particularly likely to be targeted, creating a need for effective measures to prevent fraud. Conventional fraud prevention measures have faced the challenge of being unable to detect potential fraud in real time and provide prompt countermeasures.

[0336] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0337] In this invention, the server includes a means for checking the content of emails and phone calls, a generative AI for making a prediction if the content is suspected to be special fraud, a means for advising the individual and their family about the possibility of fraud and countermeasures based on the prediction, a means for notifying the individual and their family of the advice, and a means for including the possibility of fraud and specific countermeasures in the notification. This makes it possible to detect the possibility of fraud in real time and quickly provide countermeasures.

[0338] "Means for checking the contents of emails and phone calls" refers to devices or software that analyze the contents of emails and phone calls received by the user and obtain those contents as data.

[0339] "Generative AI that makes the inference if the content is suspected to be special fraud" is an artificial intelligence system that analyzes the content of acquired emails and phone calls and infers the possibility of special fraud.

[0340] "Means for advising the individual and their family about the possibility of fraud and countermeasures based on the estimation" refers to a device or software that provides the user and their family with information about the risk of fraud and specific countermeasures based on the possibility of fraud estimated by the generative AI.

[0341] "Means for notifying the user and their family of the advice" refers to communication means for notifying the user and their family of information regarding the possibility of fraud and countermeasures.

[0342] "Means for including in the notification the possibility of fraud and specific countermeasures" refers to a device or software for including in the notification the risk of fraud and specific countermeasures.

[0343] A system for implementing this invention is configured as follows: First, a terminal such as a smartphone or PC is used as a means for a user to check the contents of emails and phone calls received. Software for analyzing the contents of emails and phone calls is installed on these terminals.

[0344] Next, a generative AI is used to make a prediction if the content is suspected to be a special fraud. This generative AI is implemented in a programming language such as Python and uses a model that has learned typical fraud patterns. Specifically, a generative AI model such as OpenAI's GPT-4 (registered trademark) is used.

[0345] Based on this prediction, the generative AI analyzes the prediction results and generates specific countermeasures to advise the individual and their family of the possibility of fraud and what countermeasures to take. These countermeasures are notified to the user and their family. Notifications are made using communication methods such as Firebase Cloud Messaging.

[0346] To include the possibility of fraud and specific countermeasures in the notification, the notification content will include the risk of fraud and specific action instructions. For example, specific advice will be provided such as, "There is a high possibility of fraud. This email contains a suspicious link. Do not provide personal information or click on the link. Please contact your bank directly for confirmation."

[0347] As a concrete example, consider the case where a user receives an email stating, "Your bank account has been fraudulently accessed. Please click this link to verify it now." The content of this email is scanned by an email analysis engine and input into a generative AI. The generative AI estimates the possibility of fraud and generates a prompt like the following:

[0348] Example prompt sentence:

[0349] "Please analyze the content of the email below and estimate the likelihood of it being a scam. If it is likely to be a scam, please provide specific countermeasures. Email content: 'Your bank account has been compromised. Click this link to check now.'"

[0350] The generative AI analyzes the prompt and generates a specific response: "This email is likely a scam. It contains a suspicious link. Do not provide any personal information or click on the link. Please contact your bank directly for confirmation." This information is then sent to the user and their family members via Firebase Cloud Messaging.

[0351] In this way, potential fraud can be detected in real time and countermeasures can be provided quickly.

[0352] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0353] Step 1:

[0354] The device acquires the contents of emails and phone calls received by the user.

[0355] Input: Content of emails and phone calls received by the user

[0356] Data processing: Extracting email and phone call content as text data

[0357] Output: Email and phone call content as text data

[0358] Step 2:

[0359] The terminal transmits the acquired text data to the server.

[0360] Input: Email and phone call content as text data

[0361] Data processing: Converting text data into a format for sending to the server

[0362] Output: Text data sent to the server

[0363] Step 3:

[0364] The server inputs the received text data into the generative AI model.

[0365] Input: Text data sent to the server

[0366] Data processing: Converting text data into a format suitable for generative AI models

[0367] Output: Text data input to the generative AI model

[0368] Step 4:

[0369] A generative AI model analyzes text data and estimates the likelihood of fraud.

[0370] Input: Text data fed into the generative AI model

[0371] Data Computing: Generative AI models analyze text data to estimate the likelihood of fraud

[0372] Output: Estimated probability of fraud

[0373] Step 5:

[0374] The server generates the probability of fraud and specific countermeasures based on the estimated results of the generated AI model.

[0375] Input: Estimated fraud probability

[0376] Data calculation: Analyzing the results of the generative AI model and generating specific countermeasures

[0377] Output: Possible fraud and specific countermeasures

[0378] Step 6:

[0379] The server sends a message to the device indicating the possibility of fraud and specific countermeasures.

[0380] Input: Possible fraud and specific countermeasures

[0381] Data processing: Converting the data into a format that can be sent to the device to identify potential fraud and specific countermeasures

[0382] Output: Possible fraud and specific countermeasures sent to the device

[0383] Step 7:

[0384] The device will notify the user and their family of possible fraud and specific countermeasures.

[0385] Input: Possible fraud and specific countermeasures sent to the terminal

[0386] Data manipulation: Formatting a notification message about potential fraud and specific countermeasures

[0387] Output: Possible fraud and specific countermeasures notified to the user and their family

[0388] In this way, potential fraud can be detected in real time and countermeasures can be provided quickly.

[0389] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0390] "Example 1"

[0391] In one embodiment of the present invention, the generative AI includes an emotion engine that recognizes the user's emotions. The emotion engine infers the user's emotional state from the user's tone of voice, facial expressions, and word choice. For example, when the user is feeling angry or confused, the emotion engine feeds that information back to the generative AI. The generative AI uses this information to adjust the accuracy of its special fraud predictions. Specifically, it can make adjustments such as strengthening the prediction of special frauds when the user is confused.

[0392] "Example 2"

[0393] In another embodiment of the present invention, the means for giving advice adjusts the content and expression of the advice according to the emotional state of the user. Specifically, when the user is panicking, the advice is given in a calm and polite manner. On the other hand, when the user is calm, the advice is given in a more direct manner. This provides advice that is appropriate for the emotional state of the user, making it easier for the user to understand the advice.

[0394] "Example 3"

[0395] In one embodiment of the present invention, the generative AI includes an emotion engine that recognizes the user's emotions. The emotion engine infers the user's emotional state from the user's tone of voice, facial expressions, and word choice. For example, when the user is feeling angry or confused, the emotion engine feeds that information back to the generative AI. The generative AI uses this information to adjust the accuracy of its special fraud predictions. Specifically, it can make adjustments such as strengthening the prediction of special frauds when the user is confused.

[0396] The processing flow of each embodiment will be described below.

[0397] "Example 1"

[0398] Step 1: Communication with the user begins. This can take various forms, such as phone calls or emails.

[0399] Step 2: The emotion engine infers the user's emotional state from their tone of voice, facial expressions, and choice of words.

[0400] Step 3: The emotion engine feeds back the estimated emotional state to the generative AI.

[0401] Step 4: The generative AI uses the feedback information to adjust the accuracy of its predictions of special frauds. Specifically, it makes adjustments such as strengthening its predictions of special frauds when the user is confused.

[0402] "Example 2"

[0403] Step 1: Communication with the user begins. This can take various forms, such as phone calls or emails.

[0404] Step 2: The emotion engine infers the user's emotional state from their tone of voice, facial expressions, and choice of words.

[0405] Step 3: The advice provider adjusts the content and expression of the advice according to the user's emotional state. Specifically, when the user is panicking, the advice is given in a calm and polite manner. On the other hand, when the user is calm, the advice is given in a more direct manner.

[0406] "Example 3"

[0407] Step 1: Communication with the user begins. This can take various forms, such as phone calls or emails.

[0408] Step 2: The emotion engine infers the user's emotional state from their tone of voice, facial expressions, and choice of words.

[0409] Step 3: The emotion engine feeds back the estimated emotional state to the generative AI.

[0410] Step 4: The generative AI uses the feedback information to adjust the accuracy of its predictions of special frauds. Specifically, it makes adjustments such as strengthening its predictions of special frauds when the user is confused.

[0411] Example 1

[0412] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0413] Conventional systems only check the content of emails and phone calls, and do not consider the user's emotional state when estimating the possibility of fraud. This makes it difficult to accurately estimate the possibility of fraud when the user is confused or emotional. Furthermore, there is a need for more accurate estimations by incorporating the user's emotional state when estimating the possibility of fraud.

[0414] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0415] In this invention, the server includes a means for checking the contents of emails and phone calls, a means for converting the contents into text, and a means for analyzing the user's emotional state, which makes it possible to estimate the likelihood of special fraud taking into account the user's emotional state.

[0416] "Means for checking the contents of emails and phone calls" refers to devices or software that acquire and analyze the contents of emails and phone calls received by the user.

[0417] The "means for converting the content into text" refers to a device or software for converting voice data or non-text data into text data.

[0418] The "means for analyzing the user's emotional state" refers to a device or software for inferring the user's emotional state from the user's tone of voice, facial expression, choice of words, etc.

[0419] "Generative AI" is an artificial intelligence model that estimates the likelihood of special fraud based on the user's emotional state and other data.

[0420] "Means for advising the user and their family of the possibility of fraud and countermeasures" refers to a device or software that notifies the user and their family of the possibility of fraud and provides specific countermeasures when there is a possibility of special fraud.

[0421] The present invention relates to a system for checking the content of emails and phone calls, analyzing the emotional state of a user, and estimating the possibility of special fraud. Specific embodiments of this system will be described below.

[0422] System configuration

[0423] How to check emails and phone calls

[0424] The server accesses the user's email account and receives new emails, and the device collects voice data in real time when the user receives a phone call, allowing the user to check the contents of emails and phone calls.

[0425] Means for converting the content into text

[0426] The server extracts the text of the received email and saves it as text data. For phone calls, the server converts the voice data into text using the Google Cloud Speech-to-Text API. This allows voice data and non-text data to be converted into text data.

[0427] A means of analyzing the user's emotional state

[0428] The server uses Librosa to analyze the tone of voice from audio data and OpenCV and Dlib to analyze facial expressions from video data, making it possible to estimate the user's emotional state from their tone of voice, facial expressions, and choice of words.

[0429] Generative AI for predicting special fraud

[0430] The server receives feedback from the emotion engine and inputs the user's emotional state into the generative AI. The generative AI estimates the likelihood of special fraud taking into account the user's emotional state. For example, if the user is confused, it adjusts the estimation accuracy to increase the likelihood of special fraud.

[0431] Advising individuals and their families about potential fraud and how to respond

[0432] The server notifies the user of the generative AI's predictions, issues warnings as necessary, and provides specific countermeasures for the user and their family in the event of possible fraud.

[0433] Specific examples

[0434] A specific example will be given below when a user receives a phone call.

[0435] 1. The user receives a call.

[0436] 2. The device collects the voice data of the call in real time.

[0437] 3. The server converts the collected voice data into text using the Google Cloud Speech-to-Text API.

[0438] 4. The server analyzes the text data and uses Librosa to analyze the tone of voice.

[0439] 5. The server analyzes the user's facial expressions using OpenCV and Dlib.

[0440] 6. The emotion engine infers that the user is confused and feeds that information back to the generative AI.

[0441] 7. The generative AI adjusts the accuracy of its estimations to increase the likelihood of special fraud.

[0442] 8. The server notifies the user of the generative AI's inference results and issues a warning if necessary.

[0443] Prompt Sentence Examples

[0444] Convert the voice data when the user receives a call into text and analyze the user's emotional state. If the user is confused, adjust the estimation accuracy to increase the possibility of special fraud.

[0445] The above is a specific embodiment for carrying out the present invention.

[0446] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0447] Step 1:

[0448] The server accesses the user's email account and receives new emails. It extracts the body of the received email and saves it as text data. The input is the email data, and the output is the text of the email body. Specifically, it connects to the email server using the IMAP protocol and retrieves the email.

[0449] Step 2:

[0450] The terminal collects voice data in real time when the user receives a call. The collected voice data is sent to the server. The input is the voice data from the call, and the output is the voice data sent to the server. Specifically, the terminal uses a microphone to capture voice and sends it to the server via the network.

[0451] Step 3:

[0452] The server converts the collected voice data into text using the Google Cloud Speech-to-Text API. The input is voice data, and the output is text-translated voice data. Specifically, the server sends the voice data to the API and saves the returned text data.

[0453] Step 4:

[0454] The server analyzes the text data and uses Librosa to analyze the tone of the voice. The input is the text audio data, and the output is the tone analysis result. Specifically, the Librosa library is used to analyze the frequency characteristics of the audio data and evaluate the tone of the voice.

[0455] Step 5:

[0456] The server analyzes the user's facial expression using OpenCV and Dlib. The input is the user's video data, and the output is the facial expression analysis results. Specifically, the server captures the user's video using a camera and analyzes the facial expression using OpenCV and Dlib.

[0457] Step 6:

[0458] The emotion engine infers that the user is confused and feeds that information back to the generative AI. The input is the results of voice tone analysis and facial expression analysis, and the output is the user's emotional state. Specifically, it integrates the analysis results and runs an algorithm to infer the user's emotional state.

[0459] Step 7:

[0460] The generative AI adjusts the estimation accuracy to increase the likelihood of special fraud. The input is the user's emotional state, and the output is the estimated result of the likelihood of special fraud. Specifically, the possibility of special fraud is estimated using a generative AI model that takes the user's emotional state into account.

[0461] Step 8:

[0462] The server notifies the user of the generative AI's inference results and issues a warning if necessary. The input is the inference result of the possibility of special fraud, and the output is a notification to the user. Specifically, it sends a warning message to the user through the notification system.

[0463] (Application example 1)

[0464] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0465] Conventional fraud prevention systems have the problem that analyzing the content of emails and phone calls alone is insufficient and do not take into account the user's emotional state, making it difficult to accurately estimate the likelihood of fraud. Furthermore, they are unable to provide appropriate countermeasures when the user is confused or angry, making it difficult to prevent fraud damage before it occurs.

[0466] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0467] In this invention, the server includes a means for checking the content of emails and phone calls, a generative AI that makes a prediction if the content is suspected to be special fraud, a means for advising the user and their family on the possibility of fraud and countermeasures based on the prediction, an emotion engine that recognizes the user's emotional state, and a means for feeding back information from the emotion engine to the generative AI and adjusting the accuracy of the prediction of special fraud. This makes it possible to accurately predict the possibility of special fraud while taking the user's emotional state into consideration and provide appropriate countermeasures.

[0468] "Means for checking the contents of emails and phone calls" refers to technology that analyzes the audio data of emails and phone calls, converts the contents into text, and checks them.

[0469] "Generative AI" is an artificial intelligence technology that learns specific patterns and characteristics based on input data and makes inferences and predictions.

[0470] "Means for advising on the possibility of fraud and countermeasures" is a technology that warns users and their families when there is a possibility of special fraud and provides specific guidelines for action.

[0471] The "emotion engine" is a technology for estimating a user's emotional state from their tone of voice, facial expression, choice of words, etc.

[0472] The "means for adjusting the estimation accuracy of special frauds" is a technology for dynamically adjusting the estimation accuracy of generative AI based on information from the emotion engine.

[0473] A system for implementing this invention is configured as follows: The server includes a means for checking the contents of emails and phone calls, a generative AI, a means for advising on the possibility of fraud and countermeasures, an emotion engine, and a means for adjusting the accuracy of special fraud estimation.

[0474] Program processing explanation

[0475] 1. How to check email and phone calls

[0476] The server uses speech recognition software (e.g., the speech_recognition library) to convert the phone call voice data into text, and can also parse the body of the email directly.

[0477] 2. Generative AI

[0478] The server uses generative AI (e.g., Pipeline from the Transformers library) to analyze the textual content and estimate the likelihood of special fraud. In this case, it uses a model that has learned typical patterns of special fraud.

[0479] 3. Advice on potential fraud and how to deal with it

[0480] Based on the generative AI's predictions, the server notifies users and their families of possible fraud and provides specific guidelines for action, such as advising them to contact the police if fraud is suspected.

[0481] 4. Emotion Engine

[0482] The server uses an emotion engine (e.g., the textblob library) to analyze the user's emotional state, inferring emotions such as anger or confusion from the user's tone of voice and word choice.

[0483] 5. Measures to adjust the accuracy of special fraud estimates

[0484] The server feeds back information from the emotion engine to the generative AI, dynamically adjusting the accuracy of its predictions for special frauds. For example, if the user is confused, it will adjust the predictions to increase the likelihood of special frauds.

[0485] Specific examples

[0486] For example, consider analyzing an audio file in which a user says "I don't know what to do" over the phone. This audio file is sent to a server and converted into text using speech recognition software. The generative AI then analyzes the text and estimates the possibility of special fraud. At the same time, the emotion engine recognizes the user's state of confusion and feeds that information back to the generative AI. Finally, the server notifies the user of the possibility of fraud and advises them on specific countermeasures.

[0487] Prompt Sentence Examples

[0488] Create a program that analyzes an audio file in which a user says "I don't know what to do" over the phone, recognizes the user's emotional state, and estimates the possibility of special fraud.

[0489] In this way, it is possible to estimate the possibility of special fraud with high accuracy while taking into account the user's emotional state and provide appropriate countermeasures.

[0490] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0491] Step 1:

[0492] The server receives the audio file sent by the user. The input is an audio file, and the output is audio data. The server converts this audio data into text using speech recognition software (e.g., the speech_recognition library). Specifically, the server reads the audio file and passes it to the speech recognition engine to convert it into text.

[0493] Step 2:

[0494] The server inputs the textual content into a generative AI. The input is text data, and the output is a predicted result indicating the likelihood of special fraud. The generative AI (e.g., pipeline from the transformers library) analyzes the text data and checks whether it contains typical patterns of special fraud. Specifically, it inputs the text data into a model and outputs the likelihood of fraud as a score.

[0495] Step 3:

[0496] The server inputs text data into the emotion engine. The input is text data, and the output is a result indicating the user's emotional state. The emotion engine (e.g., the textblob library) estimates the user's emotional state from the text data. Specifically, it inputs the text data into an emotion analysis model and outputs an emotion label (e.g., anger, confusion, joy, etc.).

[0497] Step 4:

[0498] The server feeds back emotional state information from the emotion engine to the generative AI. The input is emotional state data, and the output is the adjusted special fraud prediction result. The generative AI adjusts the prediction accuracy of special fraud based on this emotional state information. Specifically, it uses the emotional state data to recalculate the fraud probability score.

[0499] Step 5:

[0500] The server notifies the user and their family of the possibility of fraud based on the final special fraud estimation result. The input is the adjusted special fraud estimation result, and the output is a notification message. If there is a high possibility of fraud, the server sends a notification including specific guidelines for action (e.g., contact the police, do not provide personal information, etc.). Specifically, it generates a notification message and sends it to the user's device.

[0501] In this way, it is possible to estimate the possibility of special fraud with high accuracy while taking into account the user's emotional state and provide appropriate countermeasures.

[0502] Example 2

[0503] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0504] Conventional systems do not adequately detect special frauds or provide countermeasures, putting users at high risk of becoming involved in fraud. Furthermore, because they do not provide appropriate advice based on the user's emotional state, users may not be able to respond calmly. This makes it difficult to prevent fraud damage before it occurs.

[0505] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0506] In this invention, the server includes a means for checking the content of emails and phone calls, a generative AI for making a prediction if the content is suspected to be special fraud, a means for advising the user and their family on the possibility of fraud and countermeasures based on the prediction, and a means for adjusting the content and expression of the advice according to the user's emotional state. This enables early detection of special fraud and provision of appropriate advice according to the user's emotional state.

[0507] "Means for checking the contents of emails and phone calls" refers to a device or software that analyzes the contents of emails and phone calls received by the user and inputs those contents into the system.

[0508] "Generative AI" is artificial intelligence that uses machine learning models to learn specific patterns and determine whether input data matches those patterns.

[0509] "Means for advising on the possibility of fraud and countermeasures" refers to a device or software that notifies users and their families of the possibility of fraud and provides specific countermeasures based on the estimation results of the generative AI.

[0510] The "means for adjusting the content and expression of advice according to the user's emotional state" refers to a device or software that analyzes the user's emotional state and appropriately changes the content and expression of advice according to that state.

[0511] "Typical patterns of special fraud" are characteristic phrases and behavioral patterns common to special fraud, extracted from past cases.

[0512] The present invention is a system for detecting special fraud and providing advice according to the user's emotional state. A specific embodiment of this system will be described below.

[0513] Hardware and software used

[0514] Hardware:

[0515] server

[0516] Device (PC, smartphone, etc.)

[0517] software:

[0518] Machine learning models (e.g., TENSORFLOW, PyTorch)

[0519] Sentiment analysis tools (e.g., IBM Watson® Tone Analyzer)

[0520] Generative AI models (e.g., GPT-3)

[0521] System configuration

[0522] The server has the following functions:

[0523] 1. How to check email and phone calls

[0524] 2. Generative AI that makes a prediction if the content is presumed to be special fraud

[0525] 3. Based on the presumption, a means of advising the individual and their family about the possibility of fraud and countermeasures

[0526] 4. A method for adjusting the content and expression of advice according to the user's emotional state

[0527] System Operation

[0528] The user inputs the text of a received message or email into the system. For example, the user opens an application on their smartphone or PC, copies the received message, and pastes it into the input field.

[0529] The terminal sends the entered text to the server. When the terminal presses the "Send" button, the text data is sent to the server via the Internet.

[0530] The server inputs the received text into a machine learning model and analyzes whether it matches the patterns of special fraud.The server inputs the text into a machine learning model built using TensorFlow and PyTorch and scores the likelihood of fraud.

[0531] The server returns the analysis results to the device, including whether the probability of fraud is high or low. The server sends the analysis results in JSON format to the device.

[0532] The device displays the analysis results received from the server to the user. The device displays the message "This message is likely to be fraudulent" on the screen.

[0533] Advice based on emotional state

[0534] The user inputs their emotional state into the system, choosing, for example, "panic" or "calm."

[0535] The terminal transmits the user's emotional state to the server. When the terminal presses the "send" button, the emotional state data is transmitted to the server via the Internet.

[0536] The server generates appropriate advice based on the user's emotional state. A generative AI model is used to create advice based on the emotional state. The server inputs a prompt sentence into the generative AI model (e.g., GPT-3) to generate advice based on the emotional state.

[0537] The server returns the generated advice to the device. The server sends the generated advice to the device in JSON format.

[0538] The device displays the advice received from the server to the user. The device displays "Please stay calm. First, take a deep breath and stay calm." on the screen.

[0539] Specific examples

[0540] Specialized fraud detection:

[0541] Input text: "Your son has been in an accident. Please transfer the money as soon as possible."

[0542] User action: Enter text into the system and press the submit button.

[0543] Server analysis result: "Highly likely a scam."

[0544] Device display: "This message is likely a scam."

[0545] Advice based on your emotional state:

[0546] User's emotional state: Panic

[0547] User action: Select the emotional state as "Panic" and press the send button.

[0548] Server generated advice: "Stay calm. Take a deep breath and stay calm."

[0549] Device reads: "Please stay calm. First, take a deep breath and stay calm."

[0550] Example prompt sentence:

[0551] "Please determine if this message is a special scam: 'Your son has been in an accident. Please transfer money urgently.'"

[0552] "Generate advice to provide to users when they are in a panic."

[0553] "Generate advice to provide to users when they are calm."

[0554] This system allows users to detect the risk of special fraud early and take appropriate action.

[0555] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0556] Step 1:

[0557] The user inputs the text of a received message or email into the system by opening an application on their smartphone or PC, copying the received message, and pasting it into the input field.

[0558] Input: The text of your message or email

[0559] Output: Text pasted into the input field

[0560] Step 2:

[0561] The terminal sends the entered text to the server. When the terminal presses the "Send" button, the text data is sent to the server via the Internet.

[0562] Input: Text pasted into the input field

[0563] Output: Text data sent to the server

[0564] Step 3:

[0565] The server inputs the received text into a machine learning model and analyzes whether it matches the patterns of special fraud.The server inputs the text into a machine learning model built using TensorFlow and PyTorch and scores the likelihood of fraud.

[0566] Input: Text data sent to the server

[0567] Output: A fraud probability score

[0568] Step 4:

[0569] The server returns the analysis results to the device, including whether the probability of fraud is high or low. The server sends the analysis results in JSON format to the device.

[0570] Input: Fraud likelihood score

[0571] Output: Parsed results in JSON format

[0572] Step 5:

[0573] The device displays the analysis results received from the server to the user. The device displays the message "This message is likely to be fraudulent" on the screen.

[0574] Input: Parsed result in JSON format

[0575] Output: Analysis results displayed on the screen

[0576] Step 6:

[0577] The user inputs their emotional state into the system, choosing, for example, "panic" or "calm."

[0578] Input: User's emotional state

[0579] Output: Emotional state selection data

[0580] Step 7:

[0581] The terminal transmits the user's emotional state to the server. When the terminal presses the "send" button, the emotional state data is transmitted to the server via the Internet.

[0582] Input: Emotional state selection data

[0583] Output: Emotional state data sent to the server

[0584] Step 8:

[0585] The server generates appropriate advice based on the user's emotional state. A generative AI model is used to create advice based on the emotional state. The server inputs a prompt sentence into the generative AI model (e.g., GPT-3) to generate advice based on the emotional state.

[0586] Input: Emotional state data sent to the server

[0587] Output: The generated advice

[0588] Step 9:

[0589] The server returns the generated advice to the device. The server sends the generated advice to the device in JSON format.

[0590] Input: Generated advice

[0591] Output: Advice data in JSON format

[0592] Step 10:

[0593] The device displays the advice received from the server to the user. The device displays "Please stay calm. First, take a deep breath and stay calm." on the screen.

[0594] Input: Advice data in JSON format

[0595] Output: Advice displayed on the screen

[0596] (Application example 2)

[0597] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0598] In recent years, the methods of special fraud have become more sophisticated, and many people have fallen victim to them. The elderly and people with low digital literacy are particularly likely to be targeted, so effective measures to prevent fraud are needed. In addition to detecting potential fraud, it is also important to provide appropriate advice based on the user's emotional state.

[0599] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0600] In this invention, the server includes a means for checking the content of emails and phone calls, a generative AI that makes a prediction if the content is suspected to be special fraud, a means for advising the user and their family on the possibility of fraud and countermeasures based on the prediction, and a means for recognizing the user's emotional state and adjusting the content and expression of the advice depending on the emotional state. This makes it possible to detect special fraud and provide appropriate advice depending on the user's emotional state.

[0601] "Means for checking the contents of emails and phone calls" refers to a device or software that analyzes the contents of emails and phone calls received by the user and obtains those contents as text data.

[0602] "Generative AI" is an artificial intelligence system that uses machine learning models to determine whether input data matches a specific pattern.

[0603] "Means for advising on the possibility of fraud and countermeasures" refers to a device or software that notifies the user and their family of the risk of fraud and countermeasures when a possibility of special fraud is detected.

[0604] The "means for recognizing the user's emotional state" refers to a device or software that analyzes the user's voice or text data and estimates the user's emotional state.

[0605] The "means for adjusting the content and expression of advice according to the emotional state" refers to a device or software for appropriately changing the content and expression method of the advice to be provided based on the emotional state of the user.

[0606] A system for implementing this invention is configured as follows. First, the server is equipped with a means for checking the contents of emails and phone calls. This means is a device or software for acquiring and analyzing the contents of emails and phone calls received by the user as text data. Specifically, this is realized by an application installed on a terminal such as a smartphone or PC.

[0607] The server then includes a generative AI that predicts if the content is suspected to be a special fraud. This generative AI uses a machine learning model to determine whether the input text data matches a typical pattern of a special fraud. Specifically, it uses the Bidirectional Encoder Representations from Transformers (BERT) model to detect the possibility of fraud with high accuracy.

[0608] Furthermore, the server includes a means for advising the user and his / her family of the possibility of fraud and countermeasures based on the estimation. This means is a device or software for notifying the user and his / her family of the fraud risk and countermeasures when a fraud risk is detected. The notification is made via a smartphone application.

[0609] The server also includes a means for recognizing the user's emotional state. This means is a device or software that analyzes the user's voice and text data and estimates the user's emotional state. Specifically, the server uses the EmotionRecognizer library to recognize the user's emotional state in real time.

[0610] Finally, the server includes a means for adjusting the content and expression of advice according to the user's emotional state. This means is a device or software for appropriately changing the content and expression of advice provided based on the user's emotional state. For example, if the user is in a panic, a calm expression is used, and if the user is calm, a more direct expression is used.

[0611] Consider the following scenario as a concrete example: If a user receives a message saying, "Your family member has been in an accident. Please transfer money.", the system analyzes the message and determines that it is likely a special fraud. At the same time, it recognizes the user's emotional state and infers that they are in a state of panic. In this case, the system provides gentle advice such as, "Please stay calm. This may be a scam."

[0612] An example of a prompt is as follows:

[0613] Please determine whether the following text is a fraud and provide advice based on the user's emotional state.

[0614] Text: "Your relative has been in an accident. Please send some money."

[0615] User's emotional state: Panic

[0616] In this way, it is possible to detect special frauds and provide appropriate advice according to the user's emotional state.

[0617] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0618] Step 1:

[0619] The server receives emails and phone calls from the device as text data. Specifically, an application installed on the device analyzes the email or phone call content and sends the text data to the server. The input is the email or phone call content, and the output is text data.

[0620] Step 2:

[0621] The server inputs the acquired text data into a generative AI to estimate the likelihood of fraud. Specifically, it uses the BERT model to analyze the text data and determine whether it matches typical patterns of fraud. The input is text data, and the output is a flag indicating the likelihood of fraud.

[0622] Step 3:

[0623] If the server estimates that there is a high possibility of fraud, it notifies the user and their family of the risk of fraud and how to prevent it. Specifically, it sends a warning message via a smartphone application. The input is a flag indicating the possibility of fraud, and the output is a warning message.

[0624] Step 4:

[0625] The server analyzes the user's voice and text data to recognize the user's emotional state. Specifically, it uses the EmotionRecognizer library to estimate the user's emotional state in real time. The input is the user's voice and text data, and the output is data indicating the user's emotional state.

[0626] Step 5:

[0627] The server adjusts the content and expression of the advice it provides based on the user's emotional state. Specifically, it uses calm language if the user is panicked and direct language if the user is calm. The input is data indicating the user's emotional state and a flag indicating the possibility of fraud, and the output is the adjusted advice message.

[0628] Step 6:

[0629] The server sends the adjusted advice message to the user, specifically, displays the advice message through a smartphone application, where the input is the adjusted advice message and the output is a message displayed on the user's smartphone.

[0630] Example 3

[0631] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0632] Conventional special fraud prevention systems have been unable to take the user's emotional state into account when estimating the possibility of fraud, making it difficult to accurately determine the possibility of fraud. In addition, when there is a high possibility of fraud, there is a lack of means to send appropriate notifications to the person and their family, making it difficult to respond quickly and effectively.

[0633] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0634] In this invention, the server includes a means for checking the content of emails and phone calls, a generative AI that makes an inference if the content is suspected to be special fraud, a means for advising the user and their family about the possibility of fraud and countermeasures based on the inference, a means for acquiring the user's emotional state, an emotion engine that analyzes the emotional state, a means for feeding back information about the emotional state to the generative AI, a generative AI model that infers the possibility of fraud, and a means for generating and sending a notification based on the inference. This makes it possible to accurately infer the possibility of fraud while taking the user's emotional state into consideration, and to send a prompt and appropriate notification to the user and their family.

[0635] "Means for checking the contents of emails and phone calls" refers to devices or software that analyze the contents of emails and phone calls received by the user and input those contents into the system.

[0636] "Generative AI" is an artificial intelligence that estimates the possibility of special fraud based on input data.

[0637] "Means for advising the user and their family about the possibility of fraud and countermeasures" refers to devices or software that warn the user and their family about fraud and provide specific countermeasures when it is determined that there is a high possibility of fraud.

[0638] The "means for acquiring the user's emotional state" refers to a device or software for acquiring the user's emotional state from the user's voice, facial expression, etc.

[0639] The "emotion engine" is software that analyzes the acquired user emotional state and feeds the results back to the generative AI.

[0640] "Means of providing feedback to the generative AI" refers to devices or software that transmit emotional state information obtained from the emotion engine to the generative AI.

[0641] A "generative AI model" is an artificial intelligence model that estimates the possibility of special fraud based on information on emotional state.

[0642] "Means for generating and sending notifications" means devices or software that create notifications based on the output of the generative AI model and send them to the user and their family members.

[0643] This invention is a system that estimates the possibility of special fraud with high accuracy and sends prompt and appropriate notifications to the individual and their family. A specific embodiment of this system will be described below.

[0644] First, the user accesses the system using a device such as a smartphone or PC. The device accepts voice and text input so that the user can check the contents of emails and phone calls they have received. For example, the user can speak into the microphone of their smartphone and ask, "Is this email really okay?"

[0645] The device sends the user's voice data to a server in real time. The server inputs the received voice data into an emotion engine. The emotion engine uses software such as "EmotionAPI" or "Affectiva." The emotion engine analyzes the user's emotional state from the tone of the voice and choice of words, and determines that the user is confused.

[0646] The server then feeds the emotional state information obtained from the emotion engine back to the generative AI model, which uses natural language processing models such as GPT-4 and BERT. Based on this feedback, the generative AI model generates prompts to estimate the likelihood of special fraud.

[0647] The generative AI model estimates the likelihood of fraud by taking into account the user's emotional state. For example, if the user is confused, the model will set the likelihood of fraud higher. If the generative AI model determines that fraud is highly likely, it proceeds to the next step.

[0648] The server generates a notification based on the output of the generative AI model. The notification includes a warning of the possibility of fraud and specific instructions for action. For example, it might say, "You have received a suspicious email. Please refrain from providing personal information or transferring money." The server then sends this notification to the individual and any registered family members. The notification can be sent via email, SMS, or app push notification.

[0649] As a concrete example, consider a case where a user is confused. Suppose the user asks in a confused tone of voice, "Is this email really okay?" The emotion engine detects the user's confusion and feeds that information back to the generative AI model. The generative AI model then sets the likelihood of fraud high, and the server sends a notification to the user and their family saying, "You have received a suspicious email. Please refrain from providing personal information or transferring money."

[0650] Examples of prompts to input to a generative AI model might include:

[0651] User's emotional state: Confused

[0652] User says: Is this email really okay?

[0653] Generative AI model output: High probability of fraud. Send notification.

[0654] Message: You have received a suspicious email. Please refrain from providing any personal information or transferring any money.

[0655] In this way, a system is realized in which the server, terminal, and user work together to estimate the possibility of special fraud and send an appropriate notification. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0656] Step 1:

[0657] Users access the system using devices such as smartphones and PCs. To check the contents of emails and phone calls received by the user, the device accepts voice and text input. For example, a user might say into the microphone of their smartphone, "Is this email really okay?" The input is the user's voice data, and the output is the acquisition of the voice data.

[0658] Step 2:

[0659] The device sends the user's voice data to the server in real time. The server inputs the received voice data into the emotion engine. Software such as "EmotionAPI" or "Affectiva" is used for the emotion engine. The input is voice data, and the output is input to the emotion engine.

[0660] Step 3:

[0661] The emotion engine analyzes the user's emotional state from the tone of voice and word choice. The analysis result is that the user is confused. The input is voice data, and the output is the analysis result of the emotional state.

[0662] Step 4:

[0663] The server feeds back the emotional state information obtained from the emotion engine to the generative AI model. The generative AI model uses natural language processing models such as "GPT-4" and "BERT." The input is the emotional state information, and the output is feedback to the generative AI model.

[0664] Step 5:

[0665] The generative AI model generates prompts to estimate the likelihood of fraud based on emotional state information. For example, if the user is confused, the likelihood of fraud is set higher. The input is emotional state information, and the output is the estimated likelihood of fraud.

[0666] Step 6:

[0667] The server generates a notification based on the output of the generative AI model. The notification includes a warning of the possibility of fraud and specific instructions for action. For example, it might say, "You have received a suspicious email. Please avoid providing personal information or transferring money." The input is the estimated result of the possibility of fraud, and the output is the generation of a notification.

[0668] Step 7:

[0669] The server sends this notification to the person and registered family members. The notification is sent by email, SMS, app push notification, etc. The input is the generated notification, and the output is the notification sent.

[0670] (Application example 3)

[0671] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0672] In recent years, the methods used by special fraudsters have become more sophisticated, resulting in an increase in the number of cases where personal information and money are stolen via email or telephone. To prevent such fraud, a system is needed that can quickly and accurately detect potential fraud and provide appropriate countermeasures to users and their families. However, conventional systems do not take the user's emotional state into account when detecting fraud, and as a result, the accuracy of fraud estimation is sometimes insufficient. Therefore, there is a need for a system that takes the user's emotional state into account and improves the accuracy of fraud estimation.

[0673] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0674] In this invention, the server includes a means for checking the content of emails and phone calls, a generative AI that makes a prediction if the content is suspected to be special fraud, a means for advising the user and their family about the possibility of fraud and countermeasures based on the prediction, an emotion engine that recognizes the user's emotional state by analyzing the user's voice and video, a means for feeding back the emotional state to the generative AI and adjusting the accuracy of fraud prediction, and a means for sending a notification to the user and their registered family if the possibility of fraud is high. This makes it possible to improve the accuracy of fraud prediction by taking the user's emotional state into consideration.

[0675] "Means for checking the contents of emails and phone calls" refers to devices or software that analyze the contents of emails and phone calls received by the user and obtain those contents as data.

[0676] "Generative AI" is an artificial intelligence model that estimates the possibility of special fraud based on acquired data.

[0677] "Means for advising the user and their family about the possibility of fraud and countermeasures" refers to devices or software that notify the user and their family of the risk of fraud and specific countermeasures when there is a possibility of fraud.

[0678] An "emotion engine that analyzes a user's voice and video to recognize their emotional state" is a device or software that analyzes the user's voice and video data and estimates the user's emotional state.

[0679] "Means for feeding back the emotional state to the generative AI and adjusting the accuracy of fraud estimation" refers to devices or software that input the emotional state recognized by the emotion engine into the generative AI to improve the accuracy of fraud estimation.

[0680] "Means for sending a notification to the user and registered family members if there is a high possibility of fraud" refers to a device or software that sends a warning or advice to the user and their family members if there is a high possibility of fraud.

[0681] The system for implementing this invention is configured as an application installed on a user's smartphone. The system includes a means for checking the contents of emails and phone calls, a generative AI, an emotion engine, and a notification means.

[0682] First, a means for checking the contents of emails and phone calls received by the user runs on the user's smartphone, capturing the contents of emails and phone calls as text data and sending it to the generative AI for analysis.

[0683] Generative AI estimates the possibility of special fraud based on the acquired text data. This AI model learns past fraud patterns and checks whether typical patterns of fraud are included.

[0684] Next, the emotion engine analyzes the user's voice and video to recognize their emotional state. The emotion engine analyzes the user's tone of voice and facial expressions to estimate their emotional state. This emotional state is fed back to the generative AI and used to adjust the accuracy of fraud predictions.

[0685] If fraud is determined to be likely, the notification mechanism will be activated and send a notification to the user and registered family members, including information about the potential fraud and specific actions to take to protect themselves from it.

[0686] As a concrete example, consider the case where a user receives a suspicious phone call. The user enters the details of the call into the app, providing audio and video. The app analyzes this and, if it determines that the call is likely to be fraudulent, sends a notification like the one below.

[0687] Example prompt sentence:

[0688] A user receives a suspicious phone call. Analyze the audio and video to determine whether it is likely a scam. If it is, notify the user and their registered family members and advise them of specific steps to take.

[0689] This system makes it possible to improve the accuracy of fraud prediction by taking into account the user's emotional state, thereby preventing fraud from occurring.

[0690] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0691] Step 1:

[0692] A means is provided for the user to check the contents of emails and phone calls received.

[0693] Input: Contents of emails and phone calls received by the user (text data)

[0694] Output: Text data for analysis

[0695] Specific operation: The user's smartphone captures the contents of emails and phone calls as text data and sends it to the generative AI for analysis.

[0696] Step 2:

[0697] Generative AI estimates the possibility of special fraud based on the acquired text data.

[0698] Input: Text data

[0699] Output: Estimated probability of fraud (probability value)

[0700] How it works: Generative AI uses a model that has learned from past fraud patterns to check whether text data contains typical patterns of fraud and estimate the likelihood of fraud.

[0701] Step 3:

[0702] An emotion engine works by analyzing the user's voice and video to recognize their emotional state.

[0703] Input: User's audio and video data

[0704] Output: User's emotional state (e.g., confusion, anger, etc.)

[0705] Specific operation: The emotion engine analyzes the user's tone of voice and facial expressions to estimate the user's emotional state.

[0706] Step 4:

[0707] The emotional state recognized by the emotion engine is fed back to the generative AI to adjust the accuracy of fraud predictions.

[0708] Input: User's emotional state, estimated fraud probability

[0709] Output: Adjusted fraud probability estimate

[0710] Specific operation: The emotional state recognized by the emotion engine is input into the generative AI to adjust the accuracy of fraud prediction. For example, if the user is confused, the likelihood of fraud is increased.

[0711] Step 5:

[0712] If it is determined that there is a high possibility of fraud, the notification means is activated and sends a notification to the user and registered family members.

[0713] Input: Adjusted fraud probability estimate

[0714] Output: Notification to user and family (warning message and specific countermeasures)

[0715] Specific actions: If it is determined that there is a high possibility of fraud, the notification means will send a notification to the user and registered family members containing a warning message and specific countermeasures.

[0716] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0717] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0718] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0719] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0720] [Second embodiment]

[0721] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0722] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0723] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure.

[0724] The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0725] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0726] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0727] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0728] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0729] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0730] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0731] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0732] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0733] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0734] "Example 1"

[0735] In one embodiment of the present invention, software that analyzes communication data is used as a means for checking the contents of emails and phone calls. This software converts the text of emails and voice data from phone calls into text and analyzes the contents.

[0736] "Example 2"

[0737] A machine learning model is used as a generative AI to determine if the content is suspected to be special fraud. This model learns typical patterns of special fraud and determines whether the input text matches those patterns. For example, if the text contains phrases such as "Please transfer money" or "A family member of yours has been in an accident," it is determined to be highly likely to be special fraud.

[0738] "Example 3"

[0739] Based on this estimation, a notification system is used to advise the individual and their family of the possibility of fraud and countermeasures. If the system determines that there is a possibility of special fraud, it sends a notification to the individual and their registered family. The notification includes information about the possibility of fraud and specific guidelines for protecting oneself from fraud. For example, advice such as "You have received a suspicious email. Please avoid providing personal information or transferring money" is provided.

[0740] The processing flow of each embodiment will be described below.

[0741] "Example 1"

[0742] Step 1: Software that analyzes communication data converts the text of emails and audio data from phone calls into text.

[0743] Step 2: Analyze the text to see if it contains typical patterns of special fraud.

[0744] "Example 2"

[0745] Step 1: The machine learning model learns typical patterns of specialized fraud.

[0746] Step 2: Use the trained model to determine whether the input text matches a special fraud pattern.

[0747] "Example 3"

[0748] Step 1: If it is determined that there is a possibility of special fraud, the notification system will send a notification to the individual and their registered family members.

[0749] Step 2: The notice will include information about the potential fraud and specific actions you can take to protect yourself from it.

[0750] Example 1

[0751] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0752] In modern society, communication via email and telephone is on the rise, but some of this communication may contain malicious content, such as special fraud. This increases the risk of individuals and families becoming victims of fraud. Traditional methods face the challenge of quickly and accurately detecting these frauds and providing appropriate countermeasures.

[0753] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0754] In this invention, the server includes means for checking the content of emails and telephone calls, means for converting the content into text, means for analyzing the converted text, means for generating an appropriate response or action using a generative AI model based on the analysis results, a generative AI for making an inference if the content is suspected to be special fraud, and means for advising the user and their family on the possibility of fraud and countermeasures based on the inference. This makes it possible to quickly and accurately detect the possibility of special fraud from communications via emails and telephones and provide the user with appropriate countermeasures.

[0755] "Means for checking the contents of emails and phone calls" refers to a device or software that has the function of acquiring the contents of emails and phone calls received by the user and sending them to a server for analysis.

[0756] The "means for converting the content into text" refers to a device or software for converting the text of an acquired email or voice data from a telephone call into text format.

[0757] The "means for analyzing the textual content" refers to a device or software for analyzing the textual data and extracting specific keywords or phrases.

[0758] "Means for generating appropriate responses or actions using a generative AI model" refers to a device or software that uses a generative AI model to generate appropriate responses or actions for a user based on the analysis results.

[0759] "Generative AI that makes the inference when the content is suspected to be special fraud" is a generative AI model that analyzes the textual content and checks whether it contains typical patterns of special fraud.

[0760] "Means for advising the individual and their family of the possibility of fraud and countermeasures based on the presumption" refers to a device or software that notifies the user and their family of the risk of fraud and specific countermeasures when there is a possibility of special fraud.

[0761] MODE FOR CARRYING OUT THE INVENTION

[0762] The present invention relates to a system for checking the contents of emails and telephone calls to detect the possibility of special fraud. A specific embodiment of this system will be described below.

[0763] System configuration

[0764] The system consists of the following main components:

[0765] 1. Server

[0766] 2. Terminal

[0767] 3. Users

[0768] Hardware and software used

[0769] Server: A server with high-performance computing power is used. The server collects and analyzes data from mail servers and telephone systems.

[0770] Device: A device such as a computer or smartphone that a user uses.

[0771] software:

[0772] Email retrieval software: Retrieves email data using IMAP or POP3 protocols.

[0773] Speech recognition software: Converts voice data into text using software such as the Google Cloud Speech-to-Text API.

[0774] Natural Language Processing (NLP) software: Use spaCy or NLTK to analyze text data.

[0775] Generative AI models, such as OpenAI GPT-3, generate appropriate responses and actions based on analysis results.

[0776] System Operation

[0777] 1. Data Collection:

[0778] The server retrieves new emails from the user's email server, specifically by using the IMAP or POP3 protocol to collect email data.

[0779] The server retrieves call recording data from the VoIP system.

[0780] 2. Data conversion:

[0781] The server converts the collected email data into text format and extracts the text data from the email body and attachments.

[0782] The server uses voice recognition software to convert the call recording into text.

[0783] 3. Data Analysis:

[0784] The server uses natural language processing (NLP) software to analyze the text data, which analyzes the content of emails and phone calls and extracts specific keywords and phrases.

[0785] The server uses the generative AI model to generate appropriate responses and actions for the user based on the analysis results.

[0786] Specific examples

[0787] Consider the case where a user receives an email about the progress of a new project.

[0788] 1. Data Collection:

[0789] The server retrieves new mail from the user's mail server.

[0790] The server retrieves call recording data from the VoIP system.

[0791] 2. Data conversion:

[0792] The server converts the body of the email into text format.

[0793] The server uses voice recognition software to convert the call recording into text.

[0794] 3. Data Analysis:

[0795] The server analyzes the contents of the converted email and extracts the keyword "project progress."

[0796] The server analyzes the text of the call and extracts relevant information.

[0797] 4. Response Generation:

[0798] The server uses a generative AI model to input the following prompt:

[0799] Prompt: "Analyze an email about the progress of a new project and summarize the key points."

[0800] The generative AI model summarizes the contents of the email and generates a response to the user, such as "The project is progressing smoothly and the next step is ____."

[0801] In this way, the server can analyze the communication data and provide useful information to the user.

[0802] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0803] Step 1: Data collection

[0804] The server collects communication data from the user's mail server and telephone system. Specifically, it retrieves email data using the IMAP or POP3 protocol, and retrieves telephone voice data using the VoIP protocol.

[0805] Input: User's email account information, VoIP system connection information

[0806] Output: Acquired email data and audio data

[0807] Specific behavior:

[0808] The server accesses the user's email account and checks for new email.

[0809] The server connects to the VoIP system and retrieves the call recording data.

[0810] Step 2: Data conversion

[0811] The server converts the collected email data into text format, extracts text data from the email body and attachments, and uses speech recognition software (e.g., Google Cloud Speech-to-Text API) to convert voice data into text.

[0812] Input: Acquired email data and audio data

[0813] Output: Text email data, text-converted audio data

[0814] Specific behavior:

[0815] The server analyzes the body of the email and extracts the text data.

[0816] The server invokes speech recognition software to convert the voice data into text.

[0817] Step 3: Data analysis

[0818] The server uses natural language processing (NLP) software (e.g., spaCy, NLTK) to analyze the text data, which allows it to analyze the content of emails and phone calls and extract specific keywords and phrases.

[0819] Input: Text email data, audio data converted to text

[0820] Output: Extracted keywords and phrases, analysis results

[0821] Specific behavior:

[0822] The server uses NLP software to analyze the text data and extract important keywords.

[0823] The server classifies the contents of emails and phone calls based on the extracted keywords.

[0824] Step 4: Response Generation

[0825] The server uses a generative AI model (e.g., OpenAI GPT-3) to generate appropriate responses and actions for the user based on the analysis results.

[0826] Input: Analysis results, extracted keywords and phrases

[0827] Output: Response or action to the user

[0828] Specific behavior:

[0829] The server inputs the following prompt sentence into the generative AI model:

[0830] Prompt: "Analyze an email about the progress of a new project and summarize the key points."

[0831] The generative AI model summarizes the contents of the email and generates a response to the user, such as "The project is progressing smoothly and the next step is ____."

[0832] In this way, the server can analyze the communication data and provide useful information to the user.

[0833] (Application example 1)

[0834] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0835] In modern society, special frauds via email and telephone are on the rise, causing significant damage to individuals and businesses. In particular, frauds using sophisticated methods are difficult to detect using conventional methods, making it difficult to prevent damage before they occur. Furthermore, even if a possible fraud is detected, there is a lack of means to quickly provide specific countermeasures. This can prevent victims from taking appropriate measures, and the damage can become more severe. Therefore, there is a need for a system that can analyze the content of emails and telephone calls, detect possible fraud with high accuracy, and quickly provide countermeasures.

[0836] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0837] In this invention, the server includes a means for checking the content of emails and phone calls, a generative AI that makes a prediction if the content is suspected to be special fraud, a means for advising the individual and their family on the possibility of fraud and countermeasures based on the prediction, a means for converting voice data into text, a means for converting email data into text, and a means for analyzing the text data. This makes it possible to analyze the content of emails and phone calls in real time, detect the possibility of fraud with high accuracy, and quickly provide specific countermeasures.

[0838] "Means for checking the contents of emails and phone calls" refers to devices or software for analyzing the contents of received emails and phone calls and obtaining that information.

[0839] "Generative AI that makes the inference if the content is suspected to be special fraud" is artificial intelligence that analyzes the content of received emails and phone calls and determines whether the content constitutes special fraud.

[0840] "Means for advising the individual and their family of the possibility of fraud and countermeasures based on the said presumption" refers to a device or software that notifies the user and their family of the risk of fraud and countermeasures when generative AI determines that there is a possibility of special fraud.

[0841] "Means for converting voice data into text" refers to a device or software for converting voice data, such as telephone data, into text information.

[0842] "Means for converting email data into text" refers to a device or software that analyzes the content of received emails and obtains them as text information.

[0843] "Means for analyzing text data" refers to a device or software for analyzing text data and evaluating its content.

[0844] A system for implementing this invention has the following configuration: The server includes means for checking the content of emails and phone calls, a generative AI that makes an inference if the content is suspected to be special fraud, means for advising the individual and their family on the possibility of fraud and countermeasures based on the inference, means for converting voice data into text, means for converting email data into text, and means for analyzing the text data.

[0845] Hardware and Software Configuration

[0846] Hardware: Smartphone (iOS or Android)

[0847] software:

[0848] Python 3.x

[0849] speech_recognition library: Used to convert speech data into text

[0850] email library: Used to parse email data

[0851] The transformers library: used to parse text using generative AI models

[0852] Data processing and calculation

[0853] 1. Converting audio data to text:

[0854] The server uses the speech_recognition library to convert voice data, such as telephone calls, into text information, which is then used to obtain the voice data as text data.

[0855] 2. Converting email data to text:

[0856] The server uses the email library to analyze the content of the received email and obtains it as text information. This allows the email data to be obtained as text data.

[0857] 3. Text data analysis:

[0858] The server uses the transformers library to analyze the text data, evaluate its content, and use a generative AI model to determine whether the content constitutes a fraudulent activity.

[0859] 4. Possible fraud and advice on how to deal with it:

[0860] If the generative AI model determines that there is a possibility of fraud, the server notifies the user and their family of the risk of fraud and how to prevent it, providing specific guidelines for action so that the user can take appropriate action.

[0861] Specific examples

[0862] Corporate security personnel analyze the content of received emails and phone calls in real time to detect phishing scams and spam.

[0863] When individual users receive suspicious emails or phone calls, they can use this system to analyze the content and avoid security risks.

[0864] Prompt Sentence Examples

[0865] "Please analyze the contents of this email and let me know if it is a possible phishing scam."

[0866] "Please transcribe this call and let me know if it's spam."

[0867] In this way, it is possible to analyze the contents of emails and phone calls in real time, detect possible fraud with high accuracy, and quickly provide specific countermeasures.

[0868] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0869] Step 1:

[0870] The server receives email data or telephone voice data received from the user. The input is email data or voice data, and the output is the data as is. Specifically, the server sends the voice file of the email or telephone call received by the user to the server.

[0871] Step 2:

[0872] The server converts the telephone voice data into text information using a means of converting voice data to text. The input is voice data and the output is text data. Specifically, the server uses the speech_recognition library to analyze the voice data and obtain it as text information.

[0873] Step 3:

[0874] The server uses a means to convert email data into text and obtains the contents of the received email as text information. The input is email data and the output is text data. Specifically, the email library is used to parse the body of the email and obtain it as text information.

[0875] Step 4:

[0876] The server analyzes the textualized data using a means for analyzing text data. The input is text data, and the output is the analysis result. Specifically, the server uses the transformers library to input text data into a generative AI model and evaluates its content.

[0877] Step 5:

[0878] If the generative AI model determines that there is a possibility of fraud, the server notifies the user and their family of the risk of fraud and how to prevent it. The input is the analysis result, and the output is a notification message. Specifically, the server notifies the user that there is a possibility of fraud and provides specific guidelines for action.

[0879] Step 6:

[0880] The user receives a notification from the server and takes specific measures to address the risk of fraud. The input is the notification message, and the output is the user's actions. Specifically, by taking appropriate action in accordance with the notification, the user can prevent fraud before it occurs.

[0881] Example 2

[0882] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0883] In recent years, special fraud methods have become more sophisticated, and many people have fallen victim to them. Elderly people and those unfamiliar with the Internet are particularly susceptible to fraud. In order to prevent such fraud damage, a system is needed that can quickly and accurately detect possible fraud and provide appropriate countermeasures. However, current systems have low fraud detection accuracy and provide insufficient countermeasures, making it difficult to effectively prevent fraud.

[0884] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0885] In this invention, the server includes a means for a user to input text data, a means for a terminal to send the text data to the server, a means for the server to receive the text data and input it to a generative AI model, a means for the generative AI model to analyze the text data and generate a result, a means for the server to send the analysis result to the terminal, a means for the terminal to display the analysis result to the user, a generative AI that makes an inference if the content is suspected to be special fraud, and a means for advising the person and their family of the possibility of fraud and countermeasures based on the inference. This makes it possible to quickly and accurately detect the possibility of fraud and provide the user with appropriate countermeasures.

[0886] A "user" is an individual or organization that uses the system to input text data and receive analysis results.

[0887] A "terminal" is a device that allows a user to input text data, send it to a server, and display the analysis results. Specifically, this applies to smartphones and personal computers.

[0888] A "server" is a computer system that receives text data sent from a terminal, analyzes it using a generative AI model, and returns the results to the terminal.

[0889] A "generative AI model" is an artificial intelligence model that uses machine learning algorithms to learn typical patterns of special fraud and determine whether input text data matches those patterns.

[0890] "Text data" is character string information entered by the user, and is the message or sentence to be analyzed.

[0891] The "analysis result" is the judgment result of whether the text data constitutes special fraud, which is generated after the generative AI model analyzes the text data.

[0892] "Special fraud" is a type of fraud in which specific patterns or phrases are used to deceive victims and illegally obtain money or information.

[0893] "Advice tools" is a function that notifies users and their families of possible fraud based on the analysis results and provides specific guidelines for protecting themselves from fraud.

[0894] This invention is a system for detecting special frauds and providing countermeasures. The system includes a series of processes in which a user inputs text data, a terminal sends the data to a server, the server analyzes the data using a generative AI model, returns the results to the terminal, and the terminal displays the analysis results to the user.

[0895] Hardware and software used

[0896] Hardware: Servers (e.g., virtual machines for cloud computing services), devices (e.g., smartphones, personal computers)

[0897] Software: Generative AI models (e.g., GPT-3), communication protocols (e.g., HTTP / HTTPS)

[0898] Specific operation of the system

[0899] User Actions

[0900] A user inputs text data into an input field of a terminal. For example, the user inputs a message such as "Please transfer money." This message is a target for detecting the possibility of special fraud.

[0901] Device behavior

[0902] When the user presses the send button, the terminal sends the entered text data to the server. HTTP / HTTPS is used as the communication protocol. Specifically, the terminal sends a POST request to the server and includes the text data in the request body.

[0903] Server Operation

[0904] The server receives the text data sent from the device. The received text data is input into a generative AI model. For example, GPT-3 is used as the generative AI model. This model has learned typical patterns of special fraud.

[0905] How generative AI models work

[0906] The generative AI model analyzes the input text data and determines whether its content matches the pattern of specialized fraud. For example, if the message contains the phrase "Please transfer money," the model determines that the message is likely to be specialized fraud.

[0907] Server Operation

[0908] When the generative AI model generates an analysis result, the server sends the result to the terminal. Specifically, the server includes the analysis result in the response body and sends it to the terminal as an HTTP / HTTPS response.

[0909] Device behavior

[0910] The terminal displays the analysis result received from the server to the user. For example, if the analysis result indicates that "this message may be a special fraud," the terminal displays that message as a warning to the user.

[0911] Specific examples

[0912] Prompt Sentence Examples

[0913] "Please judge whether the following message is a special type of fraud: 'Please transfer money.'"

[0914] When this prompt is input into a generative AI model, the model determines whether the message matches the typical pattern of a special fraud, and returns a response such as, "This message may be a special fraud."

[0915] In this way, the system can quickly and accurately detect the possibility of special fraud and provide the user with appropriate countermeasures.

[0916] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0917] Step 1:

[0918] The user inputs text data.

[0919] A user inputs text data into an input field of a terminal. For example, the user inputs a message such as "Please transfer money." This input data is used to detect the possibility of special fraud.

[0920] Step 2:

[0921] The terminal transmits the text data to the server.

[0922] When the user presses the send button, the terminal sends the entered text data to the server. HTTP / HTTPS is used as the communication protocol. Specifically, the terminal sends a POST request to the server and includes the text data in the request body. The input is the text data entered by the user, and the output is the request sent to the server.

[0923] Step 3:

[0924] The server receives the text data and inputs it into the generative AI model.

[0925] The server receives text data sent from the device. The received text data is input into a generative AI model. For example, GPT-3 is used as the generative AI model. This model has learned typical patterns of special fraud. The input is the text data received from the device, and the output is the data input into the generative AI model.

[0926] Step 4:

[0927] A generative AI model analyzes the text data and generates results.

[0928] The generative AI model analyzes input text data and determines whether its content matches the pattern of special fraud. For example, if the message contains the phrase "Please transfer money," the model determines that it is likely to be a special fraud. The input is text data entered from the server, and the output is the analysis result.

[0929] Step 5:

[0930] The server sends the analysis results to the device.

[0931] Once the generative AI model generates the analysis results, the server sends them to the terminal. Specifically, the server includes the analysis results in the response body and sends them to the terminal as an HTTP / HTTPS response. The input is the analysis results from the generative AI model, and the output is the response sent to the terminal.

[0932] Step 6:

[0933] The terminal displays the analysis results to the user.

[0934] The terminal displays the analysis results received from the server to the user. For example, if the analysis result is "This message may be a special fraud," the terminal displays that message as a warning to the user. The input is the analysis result received from the server, and the output is the warning message displayed to the user.

[0935] (Application example 2)

[0936] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0937] In recent years, the methods of special fraud have become more sophisticated, and many people have fallen victim to them. In particular, elderly people and people with low digital literacy often fall victim to these frauds without realizing it. In order to prevent such frauds from occurring, a system that can detect potential frauds in real time and issue warnings is needed.

[0938] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for checking the content of emails and phone calls, generative AI means for making an inference if the content is inferred to be special fraud, means for advising the person and their family on the possibility of fraud and countermeasures based on the inference, and means for analyzing messages received by the user in real time and issuing a warning if there is a possibility of special fraud. This makes it possible to quickly detect the possibility of special fraud and issue a warning to the user.

[0939] "Means for checking the contents of emails and phone calls" refers to a function that allows the user to analyze the contents of emails and phone calls received and obtain those contents.

[0940] "Generative AI that makes a prediction if the content is suspected to be special fraud" is artificial intelligence that uses a machine learning model that has learned typical patterns of special fraud to determine whether the input text corresponds to special fraud.

[0941] "Means of advising the individual and their family of the possibility of fraud and countermeasures based on the estimation" refers to a function that notifies the user and their family of the possibility of fraud and provides specific countermeasures based on the results of the generative AI's estimation of special fraud.

[0942] "Means of analyzing messages received by users in real time and issuing a warning if there is a possibility of special fraud" is a function that instantly analyzes messages received by users and issues a warning if it is determined that there is a high possibility of special fraud.

[0943] The system for implementing this invention has the function of checking the contents of emails and phone calls received by the user and analyzing the possibility of special fraud in real time. The system includes the following main components.

[0944] 1. How to check emails and phone calls:

[0945] This is a function that allows users to obtain the contents of emails and phone calls they receive. This function is provided by applications installed on devices such as smartphones and computers. For example, the contents of SMS messages and emails can be automatically read.

[0946] 2. Generative AI:

[0947] This is an AI that uses a machine learning model that has learned typical patterns of special fraud to determine whether the input text corresponds to a special fraud. This generative AI uses a model built using the transformers library (e.g., distilbert-base-uncased-finetuned-sst-2-english).

[0948] 3. Advice on potential fraud and how to deal with it:

[0949] This function notifies users and their families of possible fraud based on the results of generative AI's predictions of special frauds and provides specific countermeasures. Notifications are sent via smartphone push notifications and emails.

[0950] 4. Real-time analysis and alerting measures:

[0951] This function instantly analyzes messages received by users and issues a warning if it is determined to be a special fraud. This function works the moment a user receives a message, and immediately displays a warning if there is a possibility of fraud.

[0952] Hardware and software used

[0953] Hardware: Smartphones, PCs

[0954] Software: Python, transformers library, regular expressions (re) module

[0955] Data processing and calculation

[0956] Data processing: The text entered by the user is analyzed using regular expressions to check whether it contains phrases typical of special fraud.

[0957] Data Computing: Using generative AI models to perform sentiment analysis on text to determine likelihood of fraud.

[0958] Specific examples

[0959] For example, if a user receives a message saying, "Your family member has been in an accident. Please transfer money as soon as possible," the system works as follows:

[0960] 1. Get the message content.

[0961] 2. Generative AI analyzes the message and determines whether it is a special fraud.

[0962] 3. If a fraudulent activity is deemed likely, the user will be warned.

[0963] 4. Notify users and their families of possible fraud and specific steps to take.

[0964] Prompt Sentence Examples

[0965] "Your relative has been in an accident. Please transfer the money as soon as possible."

[0966] By inputting this prompt sentence into the model, it is possible to detect possible fraud with a high probability.

[0967] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0968] Step 1:

[0969] The terminal receives messages received by the user. The input is emails or SMS messages received by the user. The terminal receives these messages in text format and passes them to the next processing step.

[0970] Step 2:

[0971] The terminal analyzes the acquired message using regular expressions. The input is the text message acquired in step 1. The terminal checks whether it contains typical phrases of special fraud, and if any such phrases are found, passes the information to the next step. The output is a flag indicating whether or not there is a possibility of fraud.

[0972] Step 3:

[0973] The server analyzes the message using a generative AI model. The input is the text message and fraud likelihood flag obtained in step 2. The server uses a generative AI model (e.g., distilbert-base-uncased-finetuned-sst-2-english) to perform sentiment analysis of the message and determine whether it is likely to be fraudulent. The output is a determination of whether it is likely to be fraudulent.

[0974] Step 4:

[0975] If the server determines that there is a high possibility of fraud, it issues a warning to the terminal. The input is the fraud possibility determination result obtained in step 3. If the server determines that there is a high possibility of fraud, it sends a warning message to the terminal. The output is a warning message that is displayed on the user's terminal.

[0976] Step 5:

[0977] The terminal notifies the user and their family of the possibility of fraud and provides specific countermeasures. The input is the warning message sent from the server in step 4. The terminal notifies the user and their family of the possibility of fraud and provides specific countermeasures. The output is the notification message displayed to the user and their family.

[0978] In this way, a system is realized that analyzes messages received by users in real time and issues a warning if there is a possibility of special fraud.

[0979] Example 3

[0980] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0981] In recent years, the methods of special fraud have become more sophisticated, and there has been an increase in cases where personal information and money are stolen via email and telephone. In order to protect individuals and their families from such fraudulent activities, a system is needed that can quickly and accurately detect possible fraud and provide appropriate countermeasures. However, current systems have low fraud detection accuracy and do not provide sufficient countermeasures, making it difficult to prevent fraud damage before it occurs.

[0982] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0983] In this invention, the server includes means for checking the content of emails and phone calls, a generative AI that makes a prediction if the content is suspected to be special fraud, means for advising the individual and their family of the possibility of fraud and countermeasures based on the prediction, means for preprocessing the content, means for inputting the preprocessed data into the generative AI, means for the generative AI to output the possibility of fraud as a score, means for generating a notification message based on the score, and means for sending the notification message to the individual and their family's devices. This makes it possible to detect the possibility of fraud with high accuracy and quickly provide appropriate countermeasures.

[0984] "Means for checking the contents of emails and phone calls" refers to devices or software that acquire and analyze the contents of emails and phone calls received by the user.

[0985] "Generative AI" is an artificial intelligence model that analyzes collected data and estimates the likelihood of fraud.

[0986] "Means for advising on the possibility of fraud and countermeasures" refers to a device or software that notifies users and their families of the risk of fraud and countermeasures based on the estimation results of the generative AI.

[0987] A "preprocessing means" is a device or software that converts collected data into a format that is easy to analyze.

[0988] "Means for inputting to generative AI" refers to a device or software for passing preprocessed data to generative AI.

[0989] A "means for outputting the likelihood of fraud as a score" is a device or software that allows generative AI to quantify the analysis results and provide the risk of fraud as a score.

[0990] A "means for generating a notification message" is a device or software for generating a warning message for the user and their family based on the fraud risk score.

[0991] The "means for transmitting a notification message to the user's and family's terminals" refers to a device or software for transmitting the generated notification message to the user's and family's devices.

[0992] This invention is a system that detects the risk of special fraud and provides appropriate countermeasures to users and their families. This system consists of multiple components, including a server, a terminal, and a generative AI model.

[0993] Hardware and software used

[0994] Server: A central server for data processing and notification sending, specifically a virtual server from a cloud service provider.

[0995] Devices: Devices on which you and your family members receive notifications, such as smartphones and tablets.

[0996] Generative AI model: An AI model for estimating the likelihood of fraud. Specifically, a generative AI model based on natural language processing is used.

[0997] Data processing and calculation

[0998] 1. Data collection: The server collects the contents of the user's emails and messages. This includes data collection with the user's consent. Specifically, it retrieves emails from the mail server using the IMAP protocol.

[0999] 2. Data preprocessing: The server preprocesses the collected data. Specifically, it cleans the text (removes unnecessary spaces and special characters). For preprocessing, a natural language processing library is used.

[1000] 3. Fraud probability estimation: The server inputs the preprocessed data into the generative AI model to estimate the likelihood of fraud. The generative AI model outputs a score indicating whether the likelihood of fraud is high.

[1001] 4. Notification Generation: If fraud is determined to be likely, the server generates a notification message stating the possibility of fraud and providing specific instructions for action.

[1002] 5. Notification sending: The server sends the generated notification to the user and registered family members' devices. The notification is sent using a messaging service.

[1003] Specific examples

[1004] User email reads: "Your account has been compromised. Please click this link to verify."

[1005] Prompt for the generative AI model: "Please determine if the following email is likely to be fraudulent. Email content: 'Your account has been compromised. Click this link to verify.'"

[1006] Notification message: "You have received a suspicious email. Please do not provide any personal information or transfer any money."

[1007] This system allows users and their families to receive specific advice on how to protect themselves from the risk of fraud. The server automatically performs a series of processes from collecting data to sending notifications, so users can avoid the risk of fraud without having to perform any special operations. The flow of the identification process in Example 3 will be described with reference to FIG. 15.

[1008] Step 1:

[1009] Data collection

[1010] A user receives an email or message.

[1011] The server retrieves new mail from the user's mail server using the IMAP protocol.

[1012] Input: User's email account information and new incoming email.

[1013] Output: The retrieved email data.

[1014] What happens: The server accesses the user's email account, downloads new emails, and temporarily stores them.

[1015] Step 2:

[1016] Data Preprocessing

[1017] Preprocessing of mail data retrieved by the server.

[1018] Input: The retrieved email data.

[1019] Output: Preprocessed text data.

[1020] What it does: The server removes unnecessary spaces and special characters from the text of the email and tokenizes the text using a natural language processing library.

[1021] Step 3:

[1022] Estimated likelihood of fraud

[1023] The server inputs the preprocessed data into the generative AI model.

[1024] Input: Preprocessed text data.

[1025] Output: Fraud likelihood score.

[1026] How it works: The server inputs the preprocessed data into the generative AI model, which outputs a fraud probability score. The generative AI model uses natural language processing techniques to analyze the text and assess the risk of fraud.

[1027] Step 4:

[1028] Notification generation

[1029] The server generates a notification message based on the fraud likelihood score.

[1030] Input: Fraud likelihood score.

[1031] Output: Informational message.

[1032] Specific Actions: If the fraud likelihood score exceeds a certain threshold, the server generates a warning message for the user and their family members, including the fraud risk and specific instructions for action.

[1033] Step 5:

[1034] Send notifications

[1035] The server generates a notification message and sends it to the terminals of the user and registered family members.

[1036] Input: Notification message.

[1037] Output: Notifications displayed on the user's and family's devices.

[1038] Specific operation: The server uses a messaging service to send the generated notification message to the user's and family's smartphones and tablets. The notification is displayed on the device so that the user and family can check it.

[1039] In this way, users and their families can receive specific advice on how to protect themselves from the risk of fraud.

[1040] (Application example 3)

[1041] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1042] In recent years, the methods used by special fraudsters have become more sophisticated, with many people falling victim to them. Elderly people and those unfamiliar with the Internet are particularly likely to be targeted, creating a need for effective measures to prevent fraud. Conventional fraud prevention measures have faced the challenge of being unable to detect potential fraud in real time and provide prompt countermeasures.

[1043] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1044] In this invention, the server includes a means for checking the content of emails and phone calls, a generative AI for making a prediction if the content is suspected to be special fraud, a means for advising the individual and their family about the possibility of fraud and countermeasures based on the prediction, a means for notifying the individual and their family of the advice, and a means for including the possibility of fraud and specific countermeasures in the notification. This makes it possible to detect the possibility of fraud in real time and quickly provide countermeasures.

[1045] "Means for checking the contents of emails and phone calls" refers to devices or software that analyze the contents of emails and phone calls received by the user and obtain those contents as data.

[1046] "Generative AI that makes the inference if the content is suspected to be special fraud" is an artificial intelligence system that analyzes the content of acquired emails and phone calls and infers the possibility of special fraud.

[1047] "Means for advising the individual and their family about the possibility of fraud and countermeasures based on the estimation" refers to a device or software that provides the user and their family with information about the risk of fraud and specific countermeasures based on the possibility of fraud estimated by the generative AI.

[1048] "Means for notifying the user and their family of the advice" refers to communication means for notifying the user and their family of information regarding the possibility of fraud and countermeasures.

[1049] "Means for including in the notification the possibility of fraud and specific countermeasures" refers to a device or software for including in the notification the risk of fraud and specific countermeasures.

[1050] A system for implementing this invention is configured as follows: First, a terminal such as a smartphone or PC is used as a means for a user to check the contents of emails and phone calls received. Software for analyzing the contents of emails and phone calls is installed on these terminals.

[1051] Next, a generative AI is used to make a prediction if the content is suspected to be a special fraud. This generative AI is implemented in a programming language such as Python and uses a model that has learned typical patterns of fraud. Specifically, a generative AI model such as OpenAI's GPT-4 is used.

[1052] Based on this prediction, the generative AI analyzes the prediction results and generates specific countermeasures to advise the individual and their family of the possibility of fraud and what countermeasures to take. These countermeasures are notified to the user and their family. Notifications are made using communication methods such as Firebase Cloud Messaging.

[1053] To include the possibility of fraud and specific countermeasures in the notification, the notification content will include the risk of fraud and specific action instructions. For example, specific advice will be provided such as, "There is a high possibility of fraud. This email contains a suspicious link. Do not provide personal information or click on the link. Please contact your bank directly for confirmation."

[1054] As a concrete example, consider the case where a user receives an email stating, "Your bank account has been fraudulently accessed. Please click this link to verify it now." The content of this email is scanned by an email analysis engine and input into a generative AI. The generative AI estimates the possibility of fraud and generates a prompt like the following:

[1055] Example prompt sentence:

[1056] "Please analyze the content of the email below and estimate the likelihood of it being a scam. If it is likely to be a scam, please provide specific countermeasures. Email content: 'Your bank account has been compromised. Click this link to check now.'"

[1057] The generative AI analyzes the prompt and generates a specific response: "This email is likely a scam. It contains a suspicious link. Do not provide any personal information or click on the link. Please contact your bank directly for confirmation." This information is then sent to the user and their family members via Firebase Cloud Messaging.

[1058] In this way, potential fraud can be detected in real time and countermeasures can be provided quickly.

[1059] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1060] Step 1:

[1061] The device acquires the contents of emails and phone calls received by the user.

[1062] Input: Content of emails and phone calls received by the user

[1063] Data processing: Extracting email and phone call content as text data

[1064] Output: Email and phone call content as text data

[1065] Step 2:

[1066] The terminal transmits the acquired text data to the server.

[1067] Input: Email and phone call content as text data

[1068] Data processing: Converting text data into a format for sending to the server

[1069] Output: Text data sent to the server

[1070] Step 3:

[1071] The server inputs the received text data into the generative AI model.

[1072] Input: Text data sent to the server

[1073] Data processing: Converting text data into a format suitable for generative AI models

[1074] Output: Text data input to the generative AI model

[1075] Step 4:

[1076] A generative AI model analyzes text data and estimates the likelihood of fraud.

[1077] Input: Text data fed into the generative AI model

[1078] Data Computing: Generative AI models analyze text data to estimate the likelihood of fraud

[1079] Output: Estimated probability of fraud

[1080] Step 5:

[1081] The server generates the probability of fraud and specific countermeasures based on the estimated results of the generated AI model.

[1082] Input: Estimated fraud probability

[1083] Data calculation: Analyzing the results of the generative AI model and generating specific countermeasures

[1084] Output: Possible fraud and specific countermeasures

[1085] Step 6:

[1086] The server sends a message to the device indicating the possibility of fraud and specific countermeasures.

[1087] Input: Possible fraud and specific countermeasures

[1088] Data processing: Converting the data into a format that can be sent to the device to identify potential fraud and specific countermeasures

[1089] Output: Possible fraud and specific countermeasures sent to the device

[1090] Step 7:

[1091] The device will notify the user and their family of possible fraud and specific countermeasures.

[1092] Input: Possible fraud and specific countermeasures sent to the terminal

[1093] Data manipulation: Formatting a notification message about potential fraud and specific countermeasures

[1094] Output: Possible fraud and specific countermeasures notified to the user and their family

[1095] In this way, potential fraud can be detected in real time and countermeasures can be provided quickly.

[1096] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1097] "Example 1"

[1098] In one embodiment of the present invention, the generative AI includes an emotion engine that recognizes the user's emotions. The emotion engine infers the user's emotional state from the user's tone of voice, facial expressions, and word choice. For example, when the user is feeling angry or confused, the emotion engine feeds that information back to the generative AI. The generative AI uses this information to adjust the accuracy of its special fraud predictions. Specifically, it can make adjustments such as strengthening the prediction of special frauds when the user is confused.

[1099] "Example 2"

[1100] In another embodiment of the present invention, the means for giving advice adjusts the content and expression of the advice according to the emotional state of the user. Specifically, when the user is panicking, the advice is given in a calm and polite manner. On the other hand, when the user is calm, the advice is given in a more direct manner. This provides advice that is appropriate for the emotional state of the user, making it easier for the user to understand the advice.

[1101] "Example 3"

[1102] In one embodiment of the present invention, the generative AI includes an emotion engine that recognizes the user's emotions. The emotion engine infers the user's emotional state from the user's tone of voice, facial expressions, and word choice. For example, when the user is feeling angry or confused, the emotion engine feeds that information back to the generative AI. The generative AI uses this information to adjust the accuracy of its special fraud predictions. Specifically, it can make adjustments such as strengthening the prediction of special frauds when the user is confused.

[1103] The processing flow of each embodiment will be described below.

[1104] "Example 1"

[1105] Step 1: Communication with the user begins. This can take various forms, such as phone calls or emails.

[1106] Step 2: The emotion engine infers the user's emotional state from their tone of voice, facial expressions, and choice of words.

[1107] Step 3: The emotion engine feeds back the estimated emotional state to the generative AI.

[1108] Step 4: The generative AI uses the feedback information to adjust the accuracy of its predictions of special frauds. Specifically, it makes adjustments such as strengthening its predictions of special frauds when the user is confused.

[1109] "Example 2"

[1110] Step 1: Communication with the user begins. This can take various forms, such as phone calls or emails.

[1111] Step 2: The emotion engine infers the user's emotional state from their tone of voice, facial expressions, and choice of words.

[1112] Step 3: The advice provider adjusts the content and expression of the advice according to the user's emotional state. Specifically, when the user is panicking, the advice is given in a calm and polite manner. On the other hand, when the user is calm, the advice is given in a more direct manner.

[1113] "Example 3"

[1114] Step 1: Communication with the user begins. This can take various forms, such as phone calls or emails.

[1115] Step 2: The emotion engine infers the user's emotional state from their tone of voice, facial expressions, and choice of words.

[1116] Step 3: The emotion engine feeds back the estimated emotional state to the generative AI.

[1117] Step 4: The generative AI uses the feedback information to adjust the accuracy of its predictions of special frauds. Specifically, it makes adjustments such as strengthening its predictions of special frauds when the user is confused.

[1118] Example 1

[1119] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1120] Conventional systems only check the content of emails and phone calls, and do not consider the user's emotional state when estimating the possibility of fraud. This makes it difficult to accurately estimate the possibility of fraud when the user is confused or emotional. Furthermore, there is a need for more accurate estimations by incorporating the user's emotional state when estimating the possibility of fraud.

[1121] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1122] In this invention, the server includes a means for checking the contents of emails and phone calls, a means for converting the contents into text, and a means for analyzing the user's emotional state, which makes it possible to estimate the likelihood of special fraud taking into account the user's emotional state.

[1123] "Means for checking the contents of emails and phone calls" refers to devices or software that acquire and analyze the contents of emails and phone calls received by the user.

[1124] The "means for converting the content into text" refers to a device or software for converting voice data or non-text data into text data.

[1125] The "means for analyzing the user's emotional state" refers to a device or software for inferring the user's emotional state from the user's tone of voice, facial expression, choice of words, etc.

[1126] "Generative AI" is an artificial intelligence model that estimates the likelihood of special fraud based on the user's emotional state and other data.

[1127] "Means for advising the user and their family of the possibility of fraud and countermeasures" refers to a device or software that notifies the user and their family of the possibility of fraud and provides specific countermeasures when there is a possibility of special fraud.

[1128] The present invention relates to a system for checking the content of emails and phone calls, analyzing the emotional state of a user, and estimating the possibility of special fraud. Specific embodiments of this system will be described below.

[1129] System configuration

[1130] How to check emails and phone calls

[1131] The server accesses the user's email account and receives new emails, and the device collects voice data in real time when the user receives a phone call, allowing the user to check the contents of emails and phone calls.

[1132] Means for converting the content into text

[1133] The server extracts the text of the received email and saves it as text data. For phone calls, the server converts the voice data into text using the Google Cloud Speech-to-Text API. This allows voice data and non-text data to be converted into text data.

[1134] A means of analyzing the user's emotional state

[1135] The server uses Librosa to analyze the tone of voice from audio data and OpenCV and Dlib to analyze facial expressions from video data, making it possible to estimate the user's emotional state from their tone of voice, facial expressions, and choice of words.

[1136] Generative AI for predicting special fraud

[1137] The server receives feedback from the emotion engine and inputs the user's emotional state into the generative AI. The generative AI estimates the likelihood of special fraud taking into account the user's emotional state. For example, if the user is confused, it adjusts the estimation accuracy to increase the likelihood of special fraud.

[1138] Advising individuals and their families about potential fraud and how to respond

[1139] The server notifies the user of the generative AI's predictions, issues warnings as necessary, and provides specific countermeasures for the user and their family in the event of possible fraud.

[1140] Specific examples

[1141] A specific example will be given below when a user receives a phone call.

[1142] 1. The user receives a call.

[1143] 2. The device collects the voice data of the call in real time.

[1144] 3. The server converts the collected voice data into text using the Google Cloud Speech-to-Text API.

[1145] 4. The server analyzes the text data and uses Librosa to analyze the tone of voice.

[1146] 5. The server analyzes the user's facial expressions using OpenCV and Dlib.

[1147] 6. The emotion engine infers that the user is confused and feeds that information back to the generative AI.

[1148] 7. The generative AI adjusts the accuracy of its estimations to increase the likelihood of special fraud.

[1149] 8. The server notifies the user of the generative AI's inference results and issues a warning if necessary.

[1150] Prompt Sentence Examples

[1151] Convert the voice data when the user receives a call into text and analyze the user's emotional state. If the user is confused, adjust the estimation accuracy to increase the possibility of special fraud.

[1152] The above is a specific embodiment for carrying out the present invention.

[1153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1154] Step 1:

[1155] The server accesses the user's email account and receives new emails. It extracts the body of the received email and saves it as text data. The input is the email data, and the output is the text of the email body. Specifically, it connects to the email server using the IMAP protocol and retrieves the email.

[1156] Step 2:

[1157] The terminal collects voice data in real time when the user receives a call. The collected voice data is sent to the server. The input is the voice data from the call, and the output is the voice data sent to the server. Specifically, the terminal uses a microphone to capture voice and sends it to the server via the network.

[1158] Step 3:

[1159] The server converts the collected voice data into text using the Google Cloud Speech-to-Text API. The input is voice data, and the output is text-translated voice data. Specifically, the server sends the voice data to the API and saves the returned text data.

[1160] Step 4:

[1161] The server analyzes the text data and uses Librosa to analyze the tone of the voice. The input is the text audio data, and the output is the tone analysis result. Specifically, the Librosa library is used to analyze the frequency characteristics of the audio data and evaluate the tone of the voice.

[1162] Step 5:

[1163] The server analyzes the user's facial expression using OpenCV and Dlib. The input is the user's video data, and the output is the facial expression analysis results. Specifically, the server captures the user's video using a camera and analyzes the facial expression using OpenCV and Dlib.

[1164] Step 6:

[1165] The emotion engine infers that the user is confused and feeds that information back to the generative AI. The input is the results of voice tone analysis and facial expression analysis, and the output is the user's emotional state. Specifically, it integrates the analysis results and runs an algorithm to infer the user's emotional state.

[1166] Step 7:

[1167] The generative AI adjusts the estimation accuracy to increase the likelihood of special fraud. The input is the user's emotional state, and the output is the estimated result of the likelihood of special fraud. Specifically, the possibility of special fraud is estimated using a generative AI model that takes the user's emotional state into account.

[1168] Step 8:

[1169] The server notifies the user of the generative AI's inference results and issues a warning if necessary. The input is the inference result of the possibility of special fraud, and the output is a notification to the user. Specifically, it sends a warning message to the user through the notification system.

[1170] (Application example 1)

[1171] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1172] Conventional fraud prevention systems have the problem that analyzing the content of emails and phone calls alone is insufficient and do not take into account the user's emotional state, making it difficult to accurately estimate the likelihood of fraud. Furthermore, they are unable to provide appropriate countermeasures when the user is confused or angry, making it difficult to prevent fraud damage before it occurs.

[1173] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1174] In this invention, the server includes a means for checking the content of emails and phone calls, a generative AI that makes a prediction if the content is suspected to be special fraud, a means for advising the user and their family on the possibility of fraud and countermeasures based on the prediction, an emotion engine that recognizes the user's emotional state, and a means for feeding back information from the emotion engine to the generative AI and adjusting the accuracy of the prediction of special fraud. This makes it possible to accurately predict the possibility of special fraud while taking the user's emotional state into consideration and provide appropriate countermeasures.

[1175] "Means for checking the contents of emails and phone calls" refers to technology that analyzes the audio data of emails and phone calls, converts the contents into text, and checks them.

[1176] "Generative AI" is an artificial intelligence technology that learns specific patterns and characteristics based on input data and makes inferences and predictions.

[1177] "Means for advising on the possibility of fraud and countermeasures" is a technology that warns users and their families when there is a possibility of special fraud and provides specific guidelines for action.

[1178] The "emotion engine" is a technology for estimating a user's emotional state from their tone of voice, facial expression, choice of words, etc.

[1179] The "means for adjusting the estimation accuracy of special frauds" is a technology for dynamically adjusting the estimation accuracy of generative AI based on information from the emotion engine.

[1180] A system for implementing this invention is configured as follows: The server includes a means for checking the contents of emails and phone calls, a generative AI, a means for advising on the possibility of fraud and countermeasures, an emotion engine, and a means for adjusting the accuracy of special fraud estimation.

[1181] Program processing explanation

[1182] 1. How to check email and phone calls

[1183] The server uses speech recognition software (e.g., the speech_recognition library) to convert the phone call voice data into text, and can also parse the body of the email directly.

[1184] 2. Generative AI

[1185] The server uses generative AI (e.g., Pipeline from the Transformers library) to analyze the textual content and estimate the likelihood of special fraud. In this case, it uses a model that has learned typical patterns of special fraud.

[1186] 3. Advice on potential fraud and how to deal with it

[1187] Based on the generative AI's predictions, the server notifies users and their families of possible fraud and provides specific guidelines for action, such as advising them to contact the police if fraud is suspected.

[1188] 4. Emotion Engine

[1189] The server uses an emotion engine (e.g., the textblob library) to analyze the user's emotional state, inferring emotions such as anger or confusion from the user's tone of voice and word choice.

[1190] 5. Measures to adjust the accuracy of special fraud estimates

[1191] The server feeds back information from the emotion engine to the generative AI, dynamically adjusting the accuracy of its predictions for special frauds. For example, if the user is confused, it will adjust the predictions to increase the likelihood of special frauds.

[1192] Specific examples

[1193] For example, consider analyzing an audio file in which a user says "I don't know what to do" over the phone. This audio file is sent to a server and converted into text using speech recognition software. The generative AI then analyzes the text and estimates the possibility of special fraud. At the same time, the emotion engine recognizes the user's state of confusion and feeds that information back to the generative AI. Finally, the server notifies the user of the possibility of fraud and advises them on specific countermeasures.

[1194] Prompt Sentence Examples

[1195] Create a program that analyzes an audio file in which a user says "I don't know what to do" over the phone, recognizes the user's emotional state, and estimates the possibility of special fraud.

[1196] In this way, it is possible to estimate the possibility of special fraud with high accuracy while taking into account the user's emotional state and provide appropriate countermeasures.

[1197] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1198] Step 1:

[1199] The server receives the audio file sent by the user. The input is an audio file, and the output is audio data. The server converts this audio data into text using speech recognition software (e.g., the speech_recognition library). Specifically, the server reads the audio file and passes it to the speech recognition engine to convert it into text.

[1200] Step 2:

[1201] The server inputs the textual content into a generative AI. The input is text data, and the output is a predicted result indicating the likelihood of special fraud. The generative AI (e.g., pipeline from the transformers library) analyzes the text data and checks whether it contains typical patterns of special fraud. Specifically, it inputs the text data into a model and outputs the likelihood of fraud as a score.

[1202] Step 3:

[1203] The server inputs text data into the emotion engine. The input is text data, and the output is a result indicating the user's emotional state. The emotion engine (e.g., the textblob library) estimates the user's emotional state from the text data. Specifically, it inputs the text data into an emotion analysis model and outputs an emotion label (e.g., anger, confusion, joy, etc.).

[1204] Step 4:

[1205] The server feeds back emotional state information from the emotion engine to the generative AI. The input is emotional state data, and the output is the adjusted special fraud prediction result. The generative AI adjusts the prediction accuracy of special fraud based on this emotional state information. Specifically, it uses the emotional state data to recalculate the fraud probability score.

[1206] Step 5:

[1207] The server notifies the user and their family of the possibility of fraud based on the final special fraud estimation result. The input is the adjusted special fraud estimation result, and the output is a notification message. If there is a high possibility of fraud, the server sends a notification including specific guidelines for action (e.g., contact the police, do not provide personal information, etc.). Specifically, it generates a notification message and sends it to the user's device.

[1208] In this way, it is possible to estimate the possibility of special fraud with high accuracy while taking into account the user's emotional state and provide appropriate countermeasures.

[1209] Example 2

[1210] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1211] Conventional systems do not adequately detect special frauds or provide countermeasures, putting users at high risk of becoming involved in fraud. Furthermore, because they do not provide appropriate advice based on the user's emotional state, users may not be able to respond calmly. This makes it difficult to prevent fraud damage before it occurs.

[1212] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1213] In this invention, the server includes a means for checking the content of emails and phone calls, a generative AI for making a prediction if the content is suspected to be special fraud, a means for advising the user and their family on the possibility of fraud and countermeasures based on the prediction, and a means for adjusting the content and expression of the advice according to the user's emotional state. This enables early detection of special fraud and provision of appropriate advice according to the user's emotional state.

[1214] "Means for checking the contents of emails and phone calls" refers to a device or software that analyzes the contents of emails and phone calls received by the user and inputs those contents into the system.

[1215] "Generative AI" is artificial intelligence that uses machine learning models to learn specific patterns and determine whether input data matches those patterns.

[1216] "Means for advising on the possibility of fraud and countermeasures" refers to a device or software that notifies users and their families of the possibility of fraud and provides specific countermeasures based on the estimation results of the generative AI.

[1217] The "means for adjusting the content and expression of advice according to the user's emotional state" refers to a device or software that analyzes the user's emotional state and appropriately changes the content and expression of advice according to that state.

[1218] "Typical patterns of special fraud" are characteristic phrases and behavioral patterns common to special fraud, extracted from past cases.

[1219] The present invention is a system for detecting special fraud and providing advice according to the user's emotional state. A specific embodiment of this system will be described below.

[1220] Hardware and software used

[1221] Hardware:

[1222] server

[1223] Device (PC, smartphone, etc.)

[1224] software:

[1225] Machine learning models (e.g., TensorFlow, PyTorch)

[1226] Sentiment analysis tools (e.g., IBM Watson Tone Analyzer)

[1227] Generative AI models (e.g., GPT-3)

[1228] System configuration

[1229] The server has the following functions:

[1230] 1. How to check email and phone calls

[1231] 2. Generative AI that makes a prediction if the content is presumed to be special fraud

[1232] 3. Based on the presumption, a means of advising the individual and their family about the possibility of fraud and countermeasures

[1233] 4. A method for adjusting the content and expression of advice according to the user's emotional state

[1234] System Operation

[1235] The user inputs the text of a received message or email into the system. For example, the user opens an application on their smartphone or PC, copies the received message, and pastes it into the input field.

[1236] The terminal sends the entered text to the server. When the terminal presses the "Send" button, the text data is sent to the server via the Internet.

[1237] The server inputs the received text into a machine learning model and analyzes whether it matches the patterns of special fraud.The server inputs the text into a machine learning model built using TensorFlow and PyTorch and scores the likelihood of fraud.

[1238] The server returns the analysis results to the device, including whether the probability of fraud is high or low. The server sends the analysis results in JSON format to the device.

[1239] The device displays the analysis results received from the server to the user. The device displays the message "This message is likely to be fraudulent" on the screen.

[1240] Advice based on emotional state

[1241] The user inputs their emotional state into the system, choosing, for example, "panic" or "calm."

[1242] The terminal transmits the user's emotional state to the server. When the terminal presses the "send" button, the emotional state data is transmitted to the server via the Internet.

[1243] The server generates appropriate advice based on the user's emotional state. A generative AI model is used to create advice based on the emotional state. The server inputs a prompt sentence into the generative AI model (e.g., GPT-3) to generate advice based on the emotional state.

[1244] The server returns the generated advice to the device. The server sends the generated advice to the device in JSON format.

[1245] The device displays the advice received from the server to the user. The device displays "Please stay calm. First, take a deep breath and stay calm." on the screen.

[1246] Specific examples

[1247] Specialized fraud detection:

[1248] Input text: "Your son has been in an accident. Please transfer the money as soon as possible."

[1249] User action: Enter text into the system and press the submit button.

[1250] Server analysis result: "Highly likely a scam."

[1251] Device display: "This message is likely a scam."

[1252] Advice based on your emotional state:

[1253] User's emotional state: Panic

[1254] User action: Select the emotional state as "Panic" and press the send button.

[1255] Server generated advice: "Stay calm. Take a deep breath and stay calm."

[1256] Device reads: "Please stay calm. First, take a deep breath and stay calm."

[1257] Example prompt sentence:

[1258] "Please determine if this message is a special scam: 'Your son has been in an accident. Please transfer money urgently.'"

[1259] "Generate advice to provide to users when they are in a panic."

[1260] "Generate advice to provide to users when they are calm."

[1261] This system allows users to detect the risk of special fraud early and take appropriate action.

[1262] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1263] Step 1:

[1264] The user inputs the text of a received message or email into the system by opening an application on their smartphone or PC, copying the received message, and pasting it into the input field.

[1265] Input: The text of your message or email

[1266] Output: Text pasted into the input field

[1267] Step 2:

[1268] The terminal sends the entered text to the server. When the terminal presses the "Send" button, the text data is sent to the server via the Internet.

[1269] Input: Text pasted into the input field

[1270] Output: Text data sent to the server

[1271] Step 3:

[1272] The server inputs the received text into a machine learning model and analyzes whether it matches the patterns of special fraud.The server inputs the text into a machine learning model built using TensorFlow and PyTorch and scores the likelihood of fraud.

[1273] Input: Text data sent to the server

[1274] Output: A fraud probability score

[1275] Step 4:

[1276] The server returns the analysis results to the device, including whether the probability of fraud is high or low. The server sends the analysis results in JSON format to the device.

[1277] Input: Fraud likelihood score

[1278] Output: Parsed results in JSON format

[1279] Step 5:

[1280] The device displays the analysis results received from the server to the user. The device displays the message "This message is likely to be fraudulent" on the screen.

[1281] Input: Parsed result in JSON format

[1282] Output: Analysis results displayed on the screen

[1283] Step 6:

[1284] The user inputs their emotional state into the system, choosing, for example, "panic" or "calm."

[1285] Input: User's emotional state

[1286] Output: Emotional state selection data

[1287] Step 7:

[1288] The terminal transmits the user's emotional state to the server. When the terminal presses the "send" button, the emotional state data is transmitted to the server via the Internet.

[1289] Input: Emotional state selection data

[1290] Output: Emotional state data sent to the server

[1291] Step 8:

[1292] The server generates appropriate advice based on the user's emotional state. A generative AI model is used to create advice based on the emotional state. The server inputs a prompt sentence into the generative AI model (e.g., GPT-3) to generate advice based on the emotional state.

[1293] Input: Emotional state data sent to the server

[1294] Output: The generated advice

[1295] Step 9:

[1296] The server returns the generated advice to the device. The server sends the generated advice to the device in JSON format.

[1297] Input: Generated advice

[1298] Output: Advice data in JSON format

[1299] Step 10:

[1300] The device displays the advice received from the server to the user. The device displays "Please stay calm. First, take a deep breath and stay calm." on the screen.

[1301] Input: Advice data in JSON format

[1302] Output: Advice displayed on the screen

[1303] (Application example 2)

[1304] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1305] In recent years, the methods of special fraud have become more sophisticated, and many people have fallen victim to them. The elderly and people with low digital literacy are particularly likely to be targeted, so effective measures to prevent fraud are needed. In addition to detecting potential fraud, it is also important to provide appropriate advice based on the user's emotional state.

[1306] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1307] In this invention, the server includes a means for checking the content of emails and phone calls, a generative AI that makes a prediction if the content is suspected to be special fraud, a means for advising the user and their family on the possibility of fraud and countermeasures based on the prediction, and a means for recognizing the user's emotional state and adjusting the content and expression of the advice depending on the emotional state. This makes it possible to detect special fraud and provide appropriate advice depending on the user's emotional state.

[1308] "Means for checking the contents of emails and phone calls" refers to a device or software that analyzes the contents of emails and phone calls received by the user and obtains those contents as text data.

[1309] "Generative AI" is an artificial intelligence system that uses machine learning models to determine whether input data matches a specific pattern.

[1310] "Means for advising on the possibility of fraud and countermeasures" refers to a device or software that notifies the user and their family of the risk of fraud and countermeasures when a possibility of special fraud is detected.

[1311] The "means for recognizing the user's emotional state" refers to a device or software that analyzes the user's voice or text data and estimates the user's emotional state.

[1312] The "means for adjusting the content and expression of advice according to the emotional state" refers to a device or software for appropriately changing the content and expression method of the advice to be provided based on the emotional state of the user.

[1313] A system for implementing this invention is configured as follows. First, the server is equipped with a means for checking the contents of emails and phone calls. This means is a device or software for acquiring and analyzing the contents of emails and phone calls received by the user as text data. Specifically, this is realized by an application installed on a terminal such as a smartphone or PC.

[1314] The server then includes a generative AI that predicts if the content is suspected to be a special fraud. This generative AI uses a machine learning model to determine whether the input text data matches a typical pattern of a special fraud. Specifically, it uses the Bidirectional Encoder Representations from Transformers (BERT) model to detect the possibility of fraud with high accuracy.

[1315] Furthermore, the server includes a means for advising the user and his / her family of the possibility of fraud and countermeasures based on the estimation. This means is a device or software for notifying the user and his / her family of the fraud risk and countermeasures when a fraud risk is detected. The notification is made via a smartphone application.

[1316] The server also includes a means for recognizing the user's emotional state. This means is a device or software that analyzes the user's voice and text data and estimates the user's emotional state. Specifically, the server uses the EmotionRecognizer library to recognize the user's emotional state in real time.

[1317] Finally, the server includes a means for adjusting the content and expression of advice according to the user's emotional state. This means is a device or software for appropriately changing the content and expression of advice provided based on the user's emotional state. For example, if the user is in a panic, a calm expression is used, and if the user is calm, a more direct expression is used.

[1318] Consider the following scenario as a concrete example: If a user receives a message saying, "Your family member has been in an accident. Please transfer money.", the system analyzes the message and determines that it is likely a special fraud. At the same time, it recognizes the user's emotional state and infers that they are in a state of panic. In this case, the system provides gentle advice such as, "Please stay calm. This may be a scam."

[1319] An example of a prompt is as follows:

[1320] Please determine whether the following text is a fraud and provide advice based on the user's emotional state.

[1321] Text: "Your relative has been in an accident. Please send some money."

[1322] User's emotional state: Panic

[1323] In this way, it is possible to detect special frauds and provide appropriate advice according to the user's emotional state.

[1324] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1325] Step 1:

[1326] The server receives emails and phone calls from the device as text data. Specifically, an application installed on the device analyzes the email or phone call content and sends the text data to the server. The input is the email or phone call content, and the output is text data.

[1327] Step 2:

[1328] The server inputs the acquired text data into a generative AI to estimate the likelihood of fraud. Specifically, it uses the BERT model to analyze the text data and determine whether it matches typical patterns of fraud. The input is text data, and the output is a flag indicating the likelihood of fraud.

[1329] Step 3:

[1330] If the server estimates that there is a high possibility of fraud, it notifies the user and their family of the risk of fraud and how to prevent it. Specifically, it sends a warning message via a smartphone application. The input is a flag indicating the possibility of fraud, and the output is a warning message.

[1331] Step 4:

[1332] The server analyzes the user's voice and text data to recognize the user's emotional state. Specifically, it uses the EmotionRecognizer library to estimate the user's emotional state in real time. The input is the user's voice and text data, and the output is data indicating the user's emotional state.

[1333] Step 5:

[1334] The server adjusts the content and expression of the advice it provides based on the user's emotional state. Specifically, it uses calm language if the user is panicked and direct language if the user is calm. The input is data indicating the user's emotional state and a flag indicating the possibility of fraud, and the output is the adjusted advice message.

[1335] Step 6:

[1336] The server sends the adjusted advice message to the user, specifically, displays the advice message through a smartphone application, where the input is the adjusted advice message and the output is a message displayed on the user's smartphone.

[1337] Example 3

[1338] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1339] Conventional special fraud prevention systems have been unable to take the user's emotional state into account when estimating the possibility of fraud, making it difficult to accurately determine the possibility of fraud. In addition, when there is a high possibility of fraud, there is a lack of means to send appropriate notifications to the person and their family, making it difficult to respond quickly and effectively.

[1340] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1341] In this invention, the server includes a means for checking the content of emails and phone calls, a generative AI that makes an inference if the content is suspected to be special fraud, a means for advising the user and their family about the possibility of fraud and countermeasures based on the inference, a means for acquiring the user's emotional state, an emotion engine that analyzes the emotional state, a means for feeding back information about the emotional state to the generative AI, a generative AI model that infers the possibility of fraud, and a means for generating and sending a notification based on the inference. This makes it possible to accurately infer the possibility of fraud while taking the user's emotional state into consideration, and to send a prompt and appropriate notification to the user and their family.

[1342] "Means for checking the contents of emails and phone calls" refers to devices or software that analyze the contents of emails and phone calls received by the user and input those contents into the system.

[1343] "Generative AI" is an artificial intelligence that estimates the possibility of special fraud based on input data.

[1344] "Means for advising the user and their family about the possibility of fraud and countermeasures" refers to devices or software that warn the user and their family about fraud and provide specific countermeasures when it is determined that there is a high possibility of fraud.

[1345] The "means for acquiring the user's emotional state" refers to a device or software for acquiring the user's emotional state from the user's voice, facial expression, etc.

[1346] The "emotion engine" is software that analyzes the acquired user emotional state and feeds the results back to the generative AI.

[1347] "Means of providing feedback to the generative AI" refers to devices or software that transmit emotional state information obtained from the emotion engine to the generative AI.

[1348] A "generative AI model" is an artificial intelligence model that estimates the possibility of special fraud based on information on emotional state.

[1349] "Means for generating and sending notifications" means devices or software that create notifications based on the output of the generative AI model and send them to the user and their family members.

[1350] This invention is a system that estimates the possibility of special fraud with high accuracy and sends prompt and appropriate notifications to the individual and their family. A specific embodiment of this system will be described below.

[1351] First, the user accesses the system using a device such as a smartphone or PC. The device accepts voice and text input so that the user can check the contents of emails and phone calls they have received. For example, the user can speak into the microphone of their smartphone and ask, "Is this email really okay?"

[1352] The device sends the user's voice data to a server in real time. The server inputs the received voice data into an emotion engine. The emotion engine uses software such as "EmotionAPI" or "Affectiva." The emotion engine analyzes the user's emotional state from the tone of the voice and choice of words, and determines that the user is confused.

[1353] The server then feeds the emotional state information obtained from the emotion engine back to the generative AI model, which uses natural language processing models such as GPT-4 and BERT. Based on this feedback, the generative AI model generates prompts to estimate the likelihood of special fraud.

[1354] The generative AI model estimates the likelihood of fraud by taking into account the user's emotional state. For example, if the user is confused, the model will set the likelihood of fraud higher. If the generative AI model determines that fraud is highly likely, it proceeds to the next step.

[1355] The server generates a notification based on the output of the generative AI model. The notification includes a warning of the possibility of fraud and specific instructions for action. For example, it might say, "You have received a suspicious email. Please refrain from providing personal information or transferring money." The server then sends this notification to the individual and any registered family members. The notification can be sent via email, SMS, or app push notification.

[1356] As a concrete example, consider a case where a user is confused. Suppose the user asks in a confused tone of voice, "Is this email really okay?" The emotion engine detects the user's confusion and feeds that information back to the generative AI model. The generative AI model then sets the likelihood of fraud high, and the server sends a notification to the user and their family saying, "You have received a suspicious email. Please refrain from providing personal information or transferring money."

[1357] Examples of prompts to input to a generative AI model might include:

[1358] User's emotional state: Confused

[1359] User says: Is this email really okay?

[1360] Generative AI model output: High probability of fraud. Send notification.

[1361] Message: You have received a suspicious email. Please refrain from providing any personal information or transferring any money.

[1362] In this way, a system is realized in which the server, terminal, and user work together to estimate the possibility of special fraud and send an appropriate notification. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1363] Step 1:

[1364] Users access the system using devices such as smartphones and PCs. To check the contents of emails and phone calls received by the user, the device accepts voice and text input. For example, a user might say into the microphone of their smartphone, "Is this email really okay?" The input is the user's voice data, and the output is the acquisition of the voice data.

[1365] Step 2:

[1366] The device sends the user's voice data to the server in real time. The server inputs the received voice data into the emotion engine. Software such as "EmotionAPI" or "Affectiva" is used for the emotion engine. The input is voice data, and the output is input to the emotion engine.

[1367] Step 3:

[1368] The emotion engine analyzes the user's emotional state from the tone of voice and word choice. The analysis result is that the user is confused. The input is voice data, and the output is the analysis result of the emotional state.

[1369] Step 4:

[1370] The server feeds back the emotional state information obtained from the emotion engine to the generative AI model. The generative AI model uses natural language processing models such as "GPT-4" and "BERT." The input is the emotional state information, and the output is feedback to the generative AI model.

[1371] Step 5:

[1372] The generative AI model generates prompts to estimate the likelihood of fraud based on emotional state information. For example, if the user is confused, the likelihood of fraud is set higher. The input is emotional state information, and the output is the estimated likelihood of fraud.

[1373] Step 6:

[1374] The server generates a notification based on the output of the generative AI model. The notification includes a warning of the possibility of fraud and specific instructions for action. For example, it might say, "You have received a suspicious email. Please avoid providing personal information or transferring money." The input is the estimated result of the possibility of fraud, and the output is the generation of a notification.

[1375] Step 7:

[1376] The server sends this notification to the person and registered family members. The notification is sent by email, SMS, app push notification, etc. The input is the generated notification, and the output is the notification sent.

[1377] (Application example 3)

[1378] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1379] In recent years, the methods used by special fraudsters have become more sophisticated, resulting in an increase in the number of cases where personal information and money are stolen via email or telephone. To prevent such fraud, a system is needed that can quickly and accurately detect potential fraud and provide appropriate countermeasures to users and their families. However, conventional systems do not take the user's emotional state into account when detecting fraud, and as a result, the accuracy of fraud estimation is sometimes insufficient. Therefore, there is a need for a system that takes the user's emotional state into account and improves the accuracy of fraud estimation.

[1380] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1381] In this invention, the server includes a means for checking the content of emails and phone calls, a generative AI that makes a prediction if the content is suspected to be special fraud, a means for advising the user and their family about the possibility of fraud and countermeasures based on the prediction, an emotion engine that recognizes the user's emotional state by analyzing the user's voice and video, a means for feeding back the emotional state to the generative AI and adjusting the accuracy of fraud prediction, and a means for sending a notification to the user and their registered family if the possibility of fraud is high. This makes it possible to improve the accuracy of fraud prediction by taking the user's emotional state into consideration.

[1382] "Means for checking the contents of emails and phone calls" refers to devices or software that analyze the contents of emails and phone calls received by the user and obtain those contents as data.

[1383] "Generative AI" is an artificial intelligence model that estimates the possibility of special fraud based on acquired data.

[1384] "Means for advising the user and their family about the possibility of fraud and countermeasures" refers to devices or software that notify the user and their family of the risk of fraud and specific countermeasures when there is a possibility of fraud.

[1385] An "emotion engine that analyzes a user's voice and video to recognize their emotional state" is a device or software that analyzes the user's voice and video data and estimates the user's emotional state.

[1386] "Means for feeding back the emotional state to the generative AI and adjusting the accuracy of fraud estimation" refers to devices or software that input the emotional state recognized by the emotion engine into the generative AI to improve the accuracy of fraud estimation.

[1387] "Means for sending a notification to the user and registered family members if there is a high possibility of fraud" refers to a device or software that sends a warning or advice to the user and their family members if there is a high possibility of fraud.

[1388] The system for implementing this invention is configured as an application installed on a user's smartphone. The system includes a means for checking the contents of emails and phone calls, a generative AI, an emotion engine, and a notification means.

[1389] First, a means for checking the contents of emails and phone calls received by the user runs on the user's smartphone, capturing the contents of emails and phone calls as text data and sending it to the generative AI for analysis.

[1390] Generative AI estimates the possibility of special fraud based on the acquired text data. This AI model learns past fraud patterns and checks whether typical patterns of fraud are included.

[1391] Next, the emotion engine analyzes the user's voice and video to recognize their emotional state. The emotion engine analyzes the user's tone of voice and facial expressions to estimate their emotional state. This emotional state is fed back to the generative AI and used to adjust the accuracy of fraud predictions.

[1392] If fraud is determined to be likely, the notification mechanism will be activated and send a notification to the user and registered family members, including information about the potential fraud and specific actions to take to protect themselves from it.

[1393] As a concrete example, consider the case where a user receives a suspicious phone call. The user enters the details of the call into the app, providing audio and video. The app analyzes this and, if it determines that the call is likely to be fraudulent, sends a notification like the one below.

[1394] Example prompt sentence:

[1395] A user receives a suspicious phone call. Analyze the audio and video to determine whether it is likely a scam. If it is, notify the user and their registered family members and advise them of specific steps to take.

[1396] This system makes it possible to improve the accuracy of fraud prediction by taking into account the user's emotional state, thereby preventing fraud from occurring.

[1397] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1398] Step 1:

[1399] A means is provided for the user to check the contents of emails and phone calls received.

[1400] Input: Contents of emails and phone calls received by the user (text data)

[1401] Output: Text data for analysis

[1402] Specific operation: The user's smartphone captures the contents of emails and phone calls as text data and sends it to the generative AI for analysis.

[1403] Step 2:

[1404] Generative AI estimates the possibility of special fraud based on the acquired text data.

[1405] Input: Text data

[1406] Output: Estimated probability of fraud (probability value)

[1407] How it works: Generative AI uses a model that has learned from past fraud patterns to check whether text data contains typical patterns of fraud and estimate the likelihood of fraud.

[1408] Step 3:

[1409] An emotion engine works by analyzing the user's voice and video to recognize their emotional state.

[1410] Input: User's audio and video data

[1411] Output: User's emotional state (e.g., confusion, anger, etc.)

[1412] Specific operation: The emotion engine analyzes the user's tone of voice and facial expressions to estimate the user's emotional state.

[1413] Step 4:

[1414] The emotional state recognized by the emotion engine is fed back to the generative AI to adjust the accuracy of fraud predictions.

[1415] Input: User's emotional state, estimated fraud probability

[1416] Output: Adjusted fraud probability estimate

[1417] Specific operation: The emotional state recognized by the emotion engine is input into the generative AI to adjust the accuracy of fraud prediction. For example, if the user is confused, the likelihood of fraud is increased.

[1418] Step 5:

[1419] If it is determined that there is a high possibility of fraud, the notification means is activated and sends a notification to the user and registered family members.

[1420] Input: Adjusted fraud probability estimate

[1421] Output: Notification to user and family (warning message and specific countermeasures)

[1422] Specific actions: If it is determined that there is a high possibility of fraud, the notification means will send a notification to the user and registered family members containing a warning message and specific countermeasures.

[1423] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1424] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1425] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1426] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1427] [Third embodiment]

[1428] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1429] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1430] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1431] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1432] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1433] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1434] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1435] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1436] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1437] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1438] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1439] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1440] "Example 1"

[1441] In one embodiment of the present invention, software that analyzes communication data is used as a means for checking the contents of emails and phone calls. This software converts the text of emails and voice data from phone calls into text and analyzes the contents.

[1442] "Example 2"

[1443] A machine learning model is used as a generative AI to determine if the content is suspected to be special fraud. This model learns typical patterns of special fraud and determines whether the input text matches those patterns. For example, if the text contains phrases such as "Please transfer money" or "A family member of yours has been in an accident," it is determined to be highly likely to be special fraud.

[1444] "Example 3"

[1445] Based on this estimation, a notification system is used to advise the individual and their family of the possibility of fraud and countermeasures. If the system determines that there is a possibility of special fraud, it sends a notification to the individual and their registered family. The notification includes information about the possibility of fraud and specific guidelines for protecting oneself from fraud. For example, advice such as "You have received a suspicious email. Please avoid providing personal information or transferring money" is provided.

[1446] The processing flow of each embodiment will be described below.

[1447] "Example 1"

[1448] Step 1: Software that analyzes communication data converts the text of emails and audio data from phone calls into text.

[1449] Step 2: Analyze the text to see if it contains typical patterns of special fraud.

[1450] "Example 2"

[1451] Step 1: The machine learning model learns typical patterns of specialized fraud.

[1452] Step 2: Use the trained model to determine whether the input text matches a special fraud pattern.

[1453] "Example 3"

[1454] Step 1: If it is determined that there is a possibility of special fraud, the notification system will send a notification to the individual and their registered family members.

[1455] Step 2: The notice will include information about the potential fraud and specific actions you can take to protect yourself from it.

[1456] Example 1

[1457] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1458] In modern society, communication via email and telephone is on the rise, but some of this communication may contain malicious content, such as special fraud. This increases the risk of individuals and families becoming victims of fraud. Traditional methods face the challenge of quickly and accurately detecting these frauds and providing appropriate countermeasures.

[1459] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1460] In this invention, the server includes means for checking the content of emails and telephone calls, means for converting the content into text, means for analyzing the converted text, means for generating an appropriate response or action using a generative AI model based on the analysis results, a generative AI for making an inference if the content is suspected to be special fraud, and means for advising the user and their family on the possibility of fraud and countermeasures based on the inference. This makes it possible to quickly and accurately detect the possibility of special fraud from communications via emails and telephones and provide the user with appropriate countermeasures.

[1461] "Means for checking the contents of emails and phone calls" refers to a device or software that has the function of acquiring the contents of emails and phone calls received by the user and sending them to a server for analysis.

[1462] The "means for converting the content into text" refers to a device or software for converting the text of an acquired email or voice data from a telephone call into text format.

[1463] The "means for analyzing the textual content" refers to a device or software for analyzing the textual data and extracting specific keywords or phrases.

[1464] "Means for generating appropriate responses or actions using a generative AI model" refers to a device or software that uses a generative AI model to generate appropriate responses or actions for a user based on the analysis results.

[1465] "Generative AI that makes the inference when the content is suspected to be special fraud" is a generative AI model that analyzes the textual content and checks whether it contains typical patterns of special fraud.

[1466] "Means for advising the individual and their family of the possibility of fraud and countermeasures based on the presumption" refers to a device or software that notifies the user and their family of the risk of fraud and specific countermeasures when there is a possibility of special fraud.

[1467] MODE FOR CARRYING OUT THE INVENTION

[1468] The present invention relates to a system for checking the contents of emails and telephone calls to detect the possibility of special fraud. A specific embodiment of this system will be described below.

[1469] System configuration

[1470] The system consists of the following main components:

[1471] 1. Server

[1472] 2. Terminal

[1473] 3. Users

[1474] Hardware and software used

[1475] Server: A server with high-performance computing power is used. The server collects and analyzes data from mail servers and telephone systems.

[1476] Device: A device such as a computer or smartphone that a user uses.

[1477] software:

[1478] Email retrieval software: Retrieves email data using IMAP or POP3 protocols.

[1479] Speech recognition software: Converts voice data into text using software such as the Google Cloud Speech-to-Text API.

[1480] Natural Language Processing (NLP) software: Use spaCy or NLTK to analyze text data.

[1481] Generative AI models, such as OpenAI GPT-3, generate appropriate responses and actions based on analysis results.

[1482] System Operation

[1483] 1. Data Collection:

[1484] The server retrieves new emails from the user's email server, specifically by using the IMAP or POP3 protocol to collect email data.

[1485] The server retrieves call recording data from the VoIP system.

[1486] 2. Data conversion:

[1487] The server converts the collected email data into text format and extracts the text data from the email body and attachments.

[1488] The server uses voice recognition software to convert the call recording into text.

[1489] 3. Data Analysis:

[1490] The server uses natural language processing (NLP) software to analyze the text data, which analyzes the content of emails and phone calls and extracts specific keywords and phrases.

[1491] The server uses the generative AI model to generate appropriate responses and actions for the user based on the analysis results.

[1492] Specific examples

[1493] Consider the case where a user receives an email about the progress of a new project.

[1494] 1. Data Collection:

[1495] The server retrieves new mail from the user's mail server.

[1496] The server retrieves call recording data from the VoIP system.

[1497] 2. Data conversion:

[1498] The server converts the body of the email into text format.

[1499] The server uses voice recognition software to convert the call recording into text.

[1500] 3. Data Analysis:

[1501] The server analyzes the contents of the converted email and extracts the keyword "project progress."

[1502] The server analyzes the text of the call and extracts relevant information.

[1503] 4. Response Generation:

[1504] The server uses a generative AI model to input the following prompt:

[1505] Prompt: "Analyze an email about the progress of a new project and summarize the key points."

[1506] The generative AI model summarizes the contents of the email and generates a response to the user, such as "The project is progressing smoothly and the next step is ____."

[1507] In this way, the server can analyze the communication data and provide useful information to the user.

[1508] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1509] Step 1: Data collection

[1510] The server collects communication data from the user's mail server and telephone system. Specifically, it retrieves email data using the IMAP or POP3 protocol, and retrieves telephone voice data using the VoIP protocol.

[1511] Input: User's email account information, VoIP system connection information

[1512] Output: Acquired email data and audio data

[1513] Specific behavior:

[1514] The server accesses the user's email account and checks for new email.

[1515] The server connects to the VoIP system and retrieves the call recording data.

[1516] Step 2: Data conversion

[1517] The server converts the collected email data into text format, extracts text data from the email body and attachments, and uses speech recognition software (e.g., Google Cloud Speech-to-Text API) to convert voice data into text.

[1518] Input: Acquired email data and audio data

[1519] Output: Text email data, text-converted audio data

[1520] Specific behavior:

[1521] The server analyzes the body of the email and extracts the text data.

[1522] The server invokes speech recognition software to convert the voice data into text.

[1523] Step 3: Data analysis

[1524] The server uses natural language processing (NLP) software (e.g., spaCy, NLTK) to analyze the text data, which allows it to analyze the content of emails and phone calls and extract specific keywords and phrases.

[1525] Input: Text email data, audio data converted to text

[1526] Output: Extracted keywords and phrases, analysis results

[1527] Specific behavior:

[1528] The server uses NLP software to analyze the text data and extract important keywords.

[1529] The server classifies the contents of emails and phone calls based on the extracted keywords.

[1530] Step 4: Response Generation

[1531] The server uses a generative AI model (e.g., OpenAI GPT-3) to generate appropriate responses and actions for the user based on the analysis results.

[1532] Input: Analysis results, extracted keywords and phrases

[1533] Output: Response or action to the user

[1534] Specific behavior:

[1535] The server inputs the following prompt sentence into the generative AI model:

[1536] Prompt: "Analyze an email about the progress of a new project and summarize the key points."

[1537] The generative AI model summarizes the contents of the email and generates a response to the user, such as "The project is progressing smoothly and the next step is ____."

[1538] In this way, the server can analyze the communication data and provide useful information to the user.

[1539] (Application example 1)

[1540] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1541] In modern society, special frauds via email and telephone are on the rise, causing significant damage to individuals and businesses. In particular, frauds using sophisticated methods are difficult to detect using conventional methods, making it difficult to prevent damage before they occur. Furthermore, even if a possible fraud is detected, there is a lack of means to quickly provide specific countermeasures. This can prevent victims from taking appropriate measures, and the damage can become more severe. Therefore, there is a need for a system that can analyze the content of emails and telephone calls, detect possible fraud with high accuracy, and quickly provide countermeasures.

[1542] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1543] In this invention, the server includes a means for checking the content of emails and phone calls, a generative AI that makes a prediction if the content is suspected to be special fraud, a means for advising the individual and their family on the possibility of fraud and countermeasures based on the prediction, a means for converting voice data into text, a means for converting email data into text, and a means for analyzing the text data. This makes it possible to analyze the content of emails and phone calls in real time, detect the possibility of fraud with high accuracy, and quickly provide specific countermeasures.

[1544] "Means for checking the contents of emails and phone calls" refers to devices or software for analyzing the contents of received emails and phone calls and obtaining that information.

[1545] "Generative AI that makes the inference if the content is suspected to be special fraud" is artificial intelligence that analyzes the content of received emails and phone calls and determines whether the content constitutes special fraud.

[1546] "Means for advising the individual and their family of the possibility of fraud and countermeasures based on the said presumption" refers to a device or software that notifies the user and their family of the risk of fraud and countermeasures when generative AI determines that there is a possibility of special fraud.

[1547] "Means for converting voice data into text" refers to a device or software for converting voice data, such as telephone data, into text information.

[1548] "Means for converting email data into text" refers to a device or software that analyzes the content of received emails and obtains them as text information.

[1549] "Means for analyzing text data" refers to a device or software for analyzing text data and evaluating its content.

[1550] A system for implementing this invention has the following configuration: The server includes means for checking the content of emails and phone calls, a generative AI that makes an inference if the content is suspected to be special fraud, means for advising the individual and their family on the possibility of fraud and countermeasures based on the inference, means for converting voice data into text, means for converting email data into text, and means for analyzing the text data.

[1551] Hardware and Software Configuration

[1552] Hardware: Smartphone (iOS or Android)

[1553] software:

[1554] Python 3.x

[1555] speech_recognition library: Used to convert speech data into text

[1556] email library: Used to parse email data

[1557] The transformers library: used to parse text using generative AI models

[1558] Data processing and calculation

[1559] 1. Converting audio data to text:

[1560] The server uses the speech_recognition library to convert voice data, such as telephone calls, into text information, which is then used to obtain the voice data as text data.

[1561] 2. Converting email data to text:

[1562] The server uses the email library to analyze the content of the received email and obtains it as text information. This allows the email data to be obtained as text data.

[1563] 3. Text data analysis:

[1564] The server uses the transformers library to analyze the text data, evaluate its content, and use a generative AI model to determine whether the content constitutes a fraudulent activity.

[1565] 4. Possible fraud and advice on how to deal with it:

[1566] If the generative AI model determines that there is a possibility of fraud, the server notifies the user and their family of the risk of fraud and how to prevent it, providing specific guidelines for action so that the user can take appropriate action.

[1567] Specific examples

[1568] Corporate security personnel analyze the content of received emails and phone calls in real time to detect phishing scams and spam.

[1569] When individual users receive suspicious emails or phone calls, they can use this system to analyze the content and avoid security risks.

[1570] Prompt Sentence Examples

[1571] "Please analyze the contents of this email and let me know if it is a possible phishing scam."

[1572] "Please transcribe this call and let me know if it's spam."

[1573] In this way, it is possible to analyze the contents of emails and phone calls in real time, detect possible fraud with high accuracy, and quickly provide specific countermeasures.

[1574] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1575] Step 1:

[1576] The server receives email data or telephone voice data received from the user. The input is email data or voice data, and the output is the data as is. Specifically, the server sends the voice file of the email or telephone call received by the user to the server.

[1577] Step 2:

[1578] The server converts the telephone voice data into text information using a means of converting voice data to text. The input is voice data and the output is text data. Specifically, the server uses the speech_recognition library to analyze the voice data and obtain it as text information.

[1579] Step 3:

[1580] The server uses a means to convert email data into text and obtains the contents of the received email as text information. The input is email data and the output is text data. Specifically, the email library is used to parse the body of the email and obtain it as text information.

[1581] Step 4:

[1582] The server analyzes the textualized data using a means for analyzing text data. The input is text data, and the output is the analysis result. Specifically, the server uses the transformers library to input text data into a generative AI model and evaluates its content.

[1583] Step 5:

[1584] If the generative AI model determines that there is a possibility of fraud, the server notifies the user and their family of the risk of fraud and how to prevent it. The input is the analysis result, and the output is a notification message. Specifically, the server notifies the user that there is a possibility of fraud and provides specific guidelines for action.

[1585] Step 6:

[1586] The user receives a notification from the server and takes specific measures to address the risk of fraud. The input is the notification message, and the output is the user's actions. Specifically, by taking appropriate action in accordance with the notification, the user can prevent fraud before it occurs.

[1587] Example 2

[1588] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1589] In recent years, special fraud methods have become more sophisticated, and many people have fallen victim to them. Elderly people and those unfamiliar with the Internet are particularly susceptible to fraud. In order to prevent such fraud damage, a system is needed that can quickly and accurately detect possible fraud and provide appropriate countermeasures. However, current systems have low fraud detection accuracy and provide insufficient countermeasures, making it difficult to effectively prevent fraud.

[1590] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1591] In this invention, the server includes a means for a user to input text data, a means for a terminal to send the text data to the server, a means for the server to receive the text data and input it to a generative AI model, a means for the generative AI model to analyze the text data and generate a result, a means for the server to send the analysis result to the terminal, a means for the terminal to display the analysis result to the user, a generative AI that makes an inference if the content is suspected to be special fraud, and a means for advising the person and their family of the possibility of fraud and countermeasures based on the inference. This makes it possible to quickly and accurately detect the possibility of fraud and provide the user with appropriate countermeasures.

[1592] A "user" is an individual or organization that uses the system to input text data and receive analysis results.

[1593] A "terminal" is a device that allows a user to input text data, send it to a server, and display the analysis results. Specifically, this applies to smartphones and personal computers.

[1594] A "server" is a computer system that receives text data sent from a terminal, analyzes it using a generative AI model, and returns the results to the terminal.

[1595] A "generative AI model" is an artificial intelligence model that uses machine learning algorithms to learn typical patterns of special fraud and determine whether input text data matches those patterns.

[1596] "Text data" is character string information entered by the user, and is the message or sentence to be analyzed.

[1597] The "analysis result" is the judgment result of whether the text data constitutes special fraud, which is generated after the generative AI model analyzes the text data.

[1598] "Special fraud" is a type of fraud in which specific patterns or phrases are used to deceive victims and illegally obtain money or information.

[1599] "Advice tools" is a function that notifies users and their families of possible fraud based on the analysis results and provides specific guidelines for protecting themselves from fraud.

[1600] This invention is a system for detecting special frauds and providing countermeasures. The system includes a series of processes in which a user inputs text data, a terminal sends the data to a server, the server analyzes the data using a generative AI model, returns the results to the terminal, and the terminal displays the analysis results to the user.

[1601] Hardware and software used

[1602] Hardware: Servers (e.g., virtual machines for cloud computing services), devices (e.g., smartphones, personal computers)

[1603] Software: Generative AI models (e.g., GPT-3), communication protocols (e.g., HTTP / HTTPS)

[1604] Specific operation of the system

[1605] User Actions

[1606] A user inputs text data into an input field of a terminal. For example, the user inputs a message such as "Please transfer money." This message is a target for detecting the possibility of special fraud.

[1607] Device behavior

[1608] When the user presses the send button, the terminal sends the entered text data to the server. HTTP / HTTPS is used as the communication protocol. Specifically, the terminal sends a POST request to the server and includes the text data in the request body.

[1609] Server Operation

[1610] The server receives the text data sent from the device. The received text data is input into a generative AI model. For example, GPT-3 is used as the generative AI model. This model has learned typical patterns of special fraud.

[1611] How generative AI models work

[1612] The generative AI model analyzes the input text data and determines whether its content matches the pattern of specialized fraud. For example, if the message contains the phrase "Please transfer money," the model determines that the message is likely to be specialized fraud.

[1613] Server Operation

[1614] When the generative AI model generates an analysis result, the server sends the result to the terminal. Specifically, the server includes the analysis result in the response body and sends it to the terminal as an HTTP / HTTPS response.

[1615] Device behavior

[1616] The terminal displays the analysis result received from the server to the user. For example, if the analysis result indicates that "this message may be a special fraud," the terminal displays that message as a warning to the user.

[1617] Specific examples

[1618] Prompt Sentence Examples

[1619] "Please judge whether the following message is a special type of fraud: 'Please transfer money.'"

[1620] When this prompt is input into a generative AI model, the model determines whether the message matches the typical pattern of a special fraud, and returns a response such as, "This message may be a special fraud."

[1621] In this way, the system can quickly and accurately detect the possibility of special fraud and provide the user with appropriate countermeasures.

[1622] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1623] Step 1:

[1624] The user inputs text data.

[1625] A user inputs text data into an input field of a terminal. For example, the user inputs a message such as "Please transfer money." This input data is used to detect the possibility of special fraud.

[1626] Step 2:

[1627] The terminal transmits the text data to the server.

[1628] When the user presses the send button, the terminal sends the entered text data to the server. HTTP / HTTPS is used as the communication protocol. Specifically, the terminal sends a POST request to the server and includes the text data in the request body. The input is the text data entered by the user, and the output is the request sent to the server.

[1629] Step 3:

[1630] The server receives the text data and inputs it into the generative AI model.

[1631] The server receives text data sent from the device. The received text data is input into a generative AI model. For example, GPT-3 is used as the generative AI model. This model has learned typical patterns of special fraud. The input is the text data received from the device, and the output is the data input into the generative AI model.

[1632] Step 4:

[1633] A generative AI model analyzes the text data and generates results.

[1634] The generative AI model analyzes input text data and determines whether its content matches the pattern of special fraud. For example, if the message contains the phrase "Please transfer money," the model determines that it is likely to be a special fraud. The input is text data entered from the server, and the output is the analysis result.

[1635] Step 5:

[1636] The server sends the analysis results to the device.

[1637] Once the generative AI model generates the analysis results, the server sends them to the terminal. Specifically, the server includes the analysis results in the response body and sends them to the terminal as an HTTP / HTTPS response. The input is the analysis results from the generative AI model, and the output is the response sent to the terminal.

[1638] Step 6:

[1639] The terminal displays the analysis results to the user.

[1640] The terminal displays the analysis results received from the server to the user. For example, if the analysis result is "This message may be a special fraud," the terminal displays that message as a warning to the user. The input is the analysis result received from the server, and the output is the warning message displayed to the user.

[1641] (Application example 2)

[1642] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1643] In recent years, the methods of special fraud have become more sophisticated, and many people have fallen victim to them. In particular, elderly people and people with low digital literacy often fall victim to these frauds without realizing it. In order to prevent such frauds from occurring, a system that can detect potential frauds in real time and issue warnings is needed.

[1644] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for checking the content of emails and phone calls, generative AI means for making an inference if the content is inferred to be special fraud, means for advising the person and their family on the possibility of fraud and countermeasures based on the inference, and means for analyzing messages received by the user in real time and issuing a warning if there is a possibility of special fraud. This makes it possible to quickly detect the possibility of special fraud and issue a warning to the user.

[1645] "Means for checking the contents of emails and phone calls" refers to a function that allows the user to analyze the contents of emails and phone calls received and obtain those contents.

[1646] "Generative AI that makes a prediction if the content is suspected to be special fraud" is artificial intelligence that uses a machine learning model that has learned typical patterns of special fraud to determine whether the input text corresponds to special fraud.

[1647] "Means of advising the individual and their family of the possibility of fraud and countermeasures based on the estimation" refers to a function that notifies the user and their family of the possibility of fraud and provides specific countermeasures based on the results of the generative AI's estimation of special fraud.

[1648] "Means of analyzing messages received by users in real time and issuing a warning if there is a possibility of special fraud" is a function that instantly analyzes messages received by users and issues a warning if it is determined that there is a high possibility of special fraud.

[1649] The system for implementing this invention has the function of checking the contents of emails and phone calls received by the user and analyzing the possibility of special fraud in real time. The system includes the following main components.

[1650] 1. How to check emails and phone calls:

[1651] This is a function that allows users to obtain the contents of emails and phone calls they receive. This function is provided by applications installed on devices such as smartphones and computers. For example, the contents of SMS messages and emails can be automatically read.

[1652] 2. Generative AI:

[1653] This is an AI that uses a machine learning model that has learned typical patterns of special fraud to determine whether the input text corresponds to a special fraud. This generative AI uses a model built using the transformers library (e.g., distilbert-base-uncased-finetuned-sst-2-english).

[1654] 3. Advice on potential fraud and how to deal with it:

[1655] This function notifies users and their families of possible fraud based on the results of generative AI's predictions of special frauds and provides specific countermeasures. Notifications are sent via smartphone push notifications and emails.

[1656] 4. Real-time analysis and alerting measures:

[1657] This function instantly analyzes messages received by users and issues a warning if it is determined to be a special fraud. This function works the moment a user receives a message, and immediately displays a warning if there is a possibility of fraud.

[1658] Hardware and software used

[1659] Hardware: Smartphones, PCs

[1660] Software: Python, transformers library, regular expressions (re) module

[1661] Data processing and calculation

[1662] Data processing: The text entered by the user is analyzed using regular expressions to check whether it contains phrases typical of special fraud.

[1663] Data Computing: Using generative AI models to perform sentiment analysis on text to determine likelihood of fraud.

[1664] Specific examples

[1665] For example, if a user receives a message saying, "Your family member has been in an accident. Please transfer money as soon as possible," the system works as follows:

[1666] 1. Get the message content.

[1667] 2. Generative AI analyzes the message and determines whether it is a special fraud.

[1668] 3. If a fraudulent activity is deemed likely, the user will be warned.

[1669] 4. Notify users and their families of possible fraud and specific steps to take.

[1670] Prompt Sentence Examples

[1671] "Your relative has been in an accident. Please transfer the money as soon as possible."

[1672] By inputting this prompt sentence into the model, it is possible to detect possible fraud with a high probability.

[1673] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1674] Step 1:

[1675] The terminal receives messages received by the user. The input is emails or SMS messages received by the user. The terminal receives these messages in text format and passes them to the next processing step.

[1676] Step 2:

[1677] The terminal analyzes the acquired message using regular expressions. The input is the text message acquired in step 1. The terminal checks whether it contains typical phrases of special fraud, and if any such phrases are found, passes the information to the next step. The output is a flag indicating whether or not there is a possibility of fraud.

[1678] Step 3:

[1679] The server analyzes the message using a generative AI model. The input is the text message and fraud likelihood flag obtained in step 2. The server uses a generative AI model (e.g., distilbert-base-uncased-finetuned-sst-2-english) to perform sentiment analysis of the message and determine whether it is likely to be fraudulent. The output is a determination of whether it is likely to be fraudulent.

[1680] Step 4:

[1681] If the server determines that there is a high possibility of fraud, it issues a warning to the terminal. The input is the fraud possibility determination result obtained in step 3. If the server determines that there is a high possibility of fraud, it sends a warning message to the terminal. The output is a warning message that is displayed on the user's terminal.

[1682] Step 5:

[1683] The terminal notifies the user and their family of the possibility of fraud and provides specific countermeasures. The input is the warning message sent from the server in step 4. The terminal notifies the user and their family of the possibility of fraud and provides specific countermeasures. The output is the notification message displayed to the user and their family.

[1684] In this way, a system is realized that analyzes messages received by users in real time and issues a warning if there is a possibility of special fraud.

[1685] Example 3

[1686] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1687] In recent years, the methods of special fraud have become more sophisticated, and there has been an increase in cases where personal information and money are stolen via email and telephone. In order to protect individuals and their families from such fraudulent activities, a system is needed that can quickly and accurately detect possible fraud and provide appropriate countermeasures. However, current systems have low fraud detection accuracy and do not provide sufficient countermeasures, making it difficult to prevent fraud damage before it occurs.

[1688] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1689] In this invention, the server includes means for checking the content of emails and phone calls, a generative AI that makes a prediction if the content is suspected to be special fraud, means for advising the individual and their family of the possibility of fraud and countermeasures based on the prediction, means for preprocessing the content, means for inputting the preprocessed data into the generative AI, means for the generative AI to output the possibility of fraud as a score, means for generating a notification message based on the score, and means for sending the notification message to the individual and their family's devices. This makes it possible to detect the possibility of fraud with high accuracy and quickly provide appropriate countermeasures.

[1690] "Means for checking the contents of emails and phone calls" refers to devices or software that acquire and analyze the contents of emails and phone calls received by the user.

[1691] "Generative AI" is an artificial intelligence model that analyzes collected data and estimates the likelihood of fraud.

[1692] "Means for advising on the possibility of fraud and countermeasures" refers to a device or software that notifies users and their families of the risk of fraud and countermeasures based on the estimation results of the generative AI.

[1693] A "preprocessing means" is a device or software that converts collected data into a format that is easy to analyze.

[1694] "Means for inputting to generative AI" refers to a device or software for passing preprocessed data to generative AI.

[1695] A "means for outputting the likelihood of fraud as a score" is a device or software that allows generative AI to quantify the analysis results and provide the risk of fraud as a score.

[1696] A "means for generating a notification message" is a device or software for generating a warning message for the user and their family based on the fraud risk score.

[1697] The "means for transmitting a notification message to the user's and family's terminals" refers to a device or software for transmitting the generated notification message to the user's and family's devices.

[1698] This invention is a system that detects the risk of special fraud and provides appropriate countermeasures to users and their families. This system consists of multiple components, including a server, a terminal, and a generative AI model.

[1699] Hardware and software used

[1700] Server: A central server for data processing and notification sending, specifically a virtual server from a cloud service provider.

[1701] Devices: Devices on which you and your family members receive notifications, such as smartphones and tablets.

[1702] Generative AI model: An AI model for estimating the likelihood of fraud. Specifically, a generative AI model based on natural language processing is used.

[1703] Data processing and calculation

[1704] 1. Data collection: The server collects the contents of the user's emails and messages. This includes data collection with the user's consent. Specifically, it retrieves emails from the mail server using the IMAP protocol.

[1705] 2. Data preprocessing: The server preprocesses the collected data. Specifically, it cleans the text (removes unnecessary spaces and special characters). For preprocessing, a natural language processing library is used.

[1706] 3. Fraud probability estimation: The server inputs the preprocessed data into the generative AI model to estimate the likelihood of fraud. The generative AI model outputs a score indicating whether the likelihood of fraud is high.

[1707] 4. Notification Generation: If fraud is determined to be likely, the server generates a notification message stating the possibility of fraud and providing specific instructions for action.

[1708] 5. Notification sending: The server sends the generated notification to the user and registered family members' devices. The notification is sent using a messaging service.

[1709] Specific examples

[1710] User email reads: "Your account has been compromised. Please click this link to verify."

[1711] Prompt for the generative AI model: "Please determine if the following email is likely to be fraudulent. Email content: 'Your account has been compromised. Click this link to verify.'"

[1712] Notification message: "You have received a suspicious email. Please do not provide any personal information or transfer any money."

[1713] This system allows users and their families to receive specific advice on how to protect themselves from the risk of fraud. The server automatically performs a series of processes from collecting data to sending notifications, so users can avoid the risk of fraud without having to perform any special operations. The flow of the identification process in Example 3 will be described with reference to FIG. 15.

[1714] Step 1:

[1715] Data collection

[1716] A user receives an email or message.

[1717] The server retrieves new mail from the user's mail server using the IMAP protocol.

[1718] Input: User's email account information and new incoming email.

[1719] Output: The retrieved email data.

[1720] What happens: The server accesses the user's email account, downloads new emails, and temporarily stores them.

[1721] Step 2:

[1722] Data Preprocessing

[1723] Preprocessing of mail data retrieved by the server.

[1724] Input: The retrieved email data.

[1725] Output: Preprocessed text data.

[1726] What it does: The server removes unnecessary spaces and special characters from the text of the email and tokenizes the text using a natural language processing library.

[1727] Step 3:

[1728] Estimated likelihood of fraud

[1729] The server inputs the preprocessed data into the generative AI model.

[1730] Input: Preprocessed text data.

[1731] Output: Fraud likelihood score.

[1732] How it works: The server inputs the preprocessed data into the generative AI model, which outputs a fraud probability score. The generative AI model uses natural language processing techniques to analyze the text and assess the risk of fraud.

[1733] Step 4:

[1734] Notification generation

[1735] The server generates a notification message based on the fraud likelihood score.

[1736] Input: Fraud likelihood score.

[1737] Output: Informational message.

[1738] Specific Actions: If the fraud likelihood score exceeds a certain threshold, the server generates a warning message for the user and their family members, including the fraud risk and specific instructions for action.

[1739] Step 5:

[1740] Send notifications

[1741] The server generates a notification message and sends it to the terminals of the user and registered family members.

[1742] Input: Notification message.

[1743] Output: Notifications displayed on the user's and family's devices.

[1744] Specific operation: The server uses a messaging service to send the generated notification message to the user's and family's smartphones and tablets. The notification is displayed on the device so that the user and family can check it.

[1745] In this way, users and their families can receive specific advice on how to protect themselves from the risk of fraud.

[1746] (Application example 3)

[1747] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1748] In recent years, the methods used by special fraudsters have become more sophisticated, with many people falling victim to them. Elderly people and those unfamiliar with the Internet are particularly likely to be targeted, creating a need for effective measures to prevent fraud. Conventional fraud prevention measures have faced the challenge of being unable to detect potential fraud in real time and provide prompt countermeasures.

[1749] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1750] In this invention, the server includes a means for checking the content of emails and phone calls, a generative AI for making a prediction if the content is suspected to be special fraud, a means for advising the individual and their family about the possibility of fraud and countermeasures based on the prediction, a means for notifying the individual and their family of the advice, and a means for including the possibility of fraud and specific countermeasures in the notification. This makes it possible to detect the possibility of fraud in real time and quickly provide countermeasures.

[1751] "Means for checking the contents of emails and phone calls" refers to devices or software that analyze the contents of emails and phone calls received by the user and obtain those contents as data.

[1752] "Generative AI that makes the inference if the content is suspected to be special fraud" is an artificial intelligence system that analyzes the content of acquired emails and phone calls and infers the possibility of special fraud.

[1753] "Means for advising the individual and their family about the possibility of fraud and countermeasures based on the estimation" refers to a device or software that provides the user and their family with information about the risk of fraud and specific countermeasures based on the possibility of fraud estimated by the generative AI.

[1754] "Means for notifying the user and their family of the advice" refers to communication means for notifying the user and their family of information regarding the possibility of fraud and countermeasures.

[1755] "Means for including in the notification the possibility of fraud and specific countermeasures" refers to a device or software for including in the notification the risk of fraud and specific countermeasures.

[1756] A system for implementing this invention is configured as follows: First, a terminal such as a smartphone or PC is used as a means for a user to check the contents of emails and phone calls received. Software for analyzing the contents of emails and phone calls is installed on these terminals.

[1757] Next, a generative AI is used to make a prediction if the content is suspected to be a special fraud. This generative AI is implemented in a programming language such as Python and uses a model that has learned typical patterns of fraud. Specifically, a generative AI model such as OpenAI's GPT-4 is used.

[1758] Based on this prediction, the generative AI analyzes the prediction results and generates specific countermeasures to advise the individual and their family of the possibility of fraud and what countermeasures to take. These countermeasures are notified to the user and their family. Notifications are made using communication methods such as Firebase Cloud Messaging.

[1759] To include the possibility of fraud and specific countermeasures in the notification, the notification content will include the risk of fraud and specific action instructions. For example, specific advice will be provided such as, "There is a high possibility of fraud. This email contains a suspicious link. Do not provide personal information or click on the link. Please contact your bank directly for confirmation."

[1760] As a concrete example, consider the case where a user receives an email stating, "Your bank account has been fraudulently accessed. Please click this link to verify it now." The content of this email is scanned by an email analysis engine and input into a generative AI. The generative AI estimates the possibility of fraud and generates a prompt like the following:

[1761] Example prompt sentence:

[1762] "Please analyze the content of the email below and estimate the likelihood of it being a scam. If it is likely to be a scam, please provide specific countermeasures. Email content: 'Your bank account has been compromised. Click this link to check now.'"

[1763] The generative AI analyzes the prompt and generates a specific response: "This email is likely a scam. It contains a suspicious link. Do not provide any personal information or click on the link. Please contact your bank directly for confirmation." This information is then sent to the user and their family members via Firebase Cloud Messaging.

[1764] In this way, potential fraud can be detected in real time and countermeasures can be provided quickly.

[1765] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1766] Step 1:

[1767] The device acquires the contents of emails and phone calls received by the user.

[1768] Input: Content of emails and phone calls received by the user

[1769] Data processing: Extracting email and phone call content as text data

[1770] Output: Email and phone call content as text data

[1771] Step 2:

[1772] The terminal transmits the acquired text data to the server.

[1773] Input: Email and phone call content as text data

[1774] Data processing: Converting text data into a format for sending to the server

[1775] Output: Text data sent to the server

[1776] Step 3:

[1777] The server inputs the received text data into the generative AI model.

[1778] Input: Text data sent to the server

[1779] Data processing: Converting text data into a format suitable for generative AI models

[1780] Output: Text data input to the generative AI model

[1781] Step 4:

[1782] A generative AI model analyzes text data and estimates the likelihood of fraud.

[1783] Input: Text data fed into the generative AI model

[1784] Data Computing: Generative AI models analyze text data to estimate the likelihood of fraud

[1785] Output: Estimated probability of fraud

[1786] Step 5:

[1787] The server generates the probability of fraud and specific countermeasures based on the estimated results of the generated AI model.

[1788] Input: Estimated fraud probability

[1789] Data calculation: Analyzing the results of the generative AI model and generating specific countermeasures

[1790] Output: Possible fraud and specific countermeasures

[1791] Step 6:

[1792] The server sends a message to the device indicating the possibility of fraud and specific countermeasures.

[1793] Input: Possible fraud and specific countermeasures

[1794] Data processing: Converting the data into a format that can be sent to the device to identify potential fraud and specific countermeasures

[1795] Output: Possible fraud and specific countermeasures sent to the device

[1796] Step 7:

[1797] The device will notify the user and their family of possible fraud and specific countermeasures.

[1798] Input: Possible fraud and specific countermeasures sent to the terminal

[1799] Data manipulation: Formatting a notification message about potential fraud and specific countermeasures

[1800] Output: Possible fraud and specific countermeasures notified to the user and their family

[1801] In this way, potential fraud can be detected in real time and countermeasures can be provided quickly.

[1802] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1803] "Example 1"

[1804] In one embodiment of the present invention, the generative AI includes an emotion engine that recognizes a user's emotions by inferring their emotional state from their tone of voice, facial expressions, and word choice.

[1805] For example, when a user feels angry or confused, the emotion engine feeds that information back to the generative AI. The generative AI uses this information to adjust the accuracy of its predictions of special frauds. Specifically, it can make adjustments such as strengthening predictions of special frauds when the user is confused.

[1806] "Example 2"

[1807] In another embodiment of the present invention, the means for giving advice adjusts the content and expression of the advice according to the emotional state of the user. Specifically, when the user is panicking, the advice is given in a calm and polite manner. On the other hand, when the user is calm, the advice is given in a more direct manner. This provides advice that is appropriate for the emotional state of the user, making it easier for the user to understand the advice.

[1808] "Example 3"

[1809] In one embodiment of the present invention, the generative AI includes an emotion engine that recognizes a user's emotions by inferring their emotional state from their tone of voice, facial expressions, and word choice.

[1810] For example, when a user feels angry or confused, the emotion engine feeds that information back to the generative AI. The generative AI uses this information to adjust the accuracy of its predictions of special frauds. Specifically, it can make adjustments such as strengthening predictions of special frauds when the user is confused.

[1811] The processing flow of each embodiment will be described below.

[1812] "Example 1"

[1813] Step 1: Communication with the user begins. This can take various forms, such as phone calls or emails.

[1814] Step 2: The emotion engine infers the user's emotional state from their tone of voice, facial expressions, and choice of words.

[1815] Step 3: The emotion engine feeds back the estimated emotional state to the generative AI.

[1816] Step 4: The generative AI uses the feedback information to adjust the accuracy of its predictions of special frauds. Specifically, it makes adjustments such as strengthening its predictions of special frauds when the user is confused.

[1817] "Example 2"

[1818] Step 1: Communication with the user begins. This can take various forms, such as phone calls or emails.

[1819] Step 2: The emotion engine infers the user's emotional state from their tone of voice, facial expressions, and choice of words.

[1820] Step 3: The advice provider adjusts the content and expression of the advice according to the user's emotional state. Specifically, when the user is panicking, the advice is given in a calm and polite manner. On the other hand, when the user is calm, the advice is given in a more direct manner.

[1821] "Example 3"

[1822] Step 1: Communication with the user begins. This can take various forms, such as phone calls or emails.

[1823] Step 2: The emotion engine infers the user's emotional state from their tone of voice, facial expressions, and choice of words.

[1824] Step 3: The emotion engine feeds back the estimated emotional state to the generative AI.

[1825] Step 4: The generative AI uses the feedback information to adjust the accuracy of its predictions of special frauds. Specifically, it makes adjustments such as strengthening its predictions of special frauds when the user is confused.

[1826] Example 1

[1827] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1828] Conventional systems only check the content of emails and phone calls, and do not consider the user's emotional state when estimating the possibility of fraud. This makes it difficult to accurately estimate the possibility of fraud when the user is confused or emotional. Furthermore, there is a need for more accurate estimations by incorporating the user's emotional state when estimating the possibility of fraud.

[1829] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1830] In this invention, the server includes a means for checking the contents of emails and phone calls, a means for converting the contents into text, and a means for analyzing the user's emotional state, which makes it possible to estimate the likelihood of special fraud taking into account the user's emotional state.

[1831] "Means for checking the contents of emails and phone calls" refers to devices or software that acquire and analyze the contents of emails and phone calls received by the user.

[1832] The "means for converting the content into text" refers to a device or software for converting voice data or non-text data into text data.

[1833] The "means for analyzing the user's emotional state" refers to a device or software for inferring the user's emotional state from the user's tone of voice, facial expression, choice of words, etc.

[1834] "Generative AI" is an artificial intelligence model that estimates the likelihood of special fraud based on the user's emotional state and other data.

[1835] "Means for advising the user and their family of the possibility of fraud and countermeasures" refers to a device or software that notifies the user and their family of the possibility of fraud and provides specific countermeasures when there is a possibility of special fraud.

[1836] The present invention relates to a system for checking the content of emails and phone calls, analyzing the emotional state of a user, and estimating the possibility of special fraud. Specific embodiments of this system will be described below.

[1837] System configuration

[1838] How to check emails and phone calls

[1839] The server accesses the user's email account and receives new emails, and the device collects voice data in real time when the user receives a phone call, allowing the user to check the contents of emails and phone calls.

[1840] Means for converting the content into text

[1841] The server extracts the text of the received email and saves it as text data. For phone calls, the server converts the voice data into text using the Google Cloud Speech-to-Text API. This allows voice data and non-text data to be converted into text data.

[1842] A means of analyzing the user's emotional state

[1843] The server uses Librosa to analyze the tone of voice from audio data and OpenCV and Dlib to analyze facial expressions from video data, making it possible to estimate the user's emotional state from their tone of voice, facial expressions, and choice of words.

[1844] Generative AI for predicting special fraud

[1845] The server receives feedback from the emotion engine and inputs the user's emotional state into the generative AI. The generative AI estimates the likelihood of special fraud taking into account the user's emotional state. For example, if the user is confused, it adjusts the estimation accuracy to increase the likelihood of special fraud.

[1846] Advising individuals and their families about potential fraud and how to respond

[1847] The server notifies the user of the generative AI's predictions, issues warnings as necessary, and provides specific countermeasures for the user and their family in the event of possible fraud.

[1848] Specific examples

[1849] A specific example will be given below when a user receives a phone call.

[1850] 1. The user receives a call.

[1851] 2. The device collects the voice data of the call in real time.

[1852] 3. The server converts the collected voice data into text using the Google Cloud Speech-to-Text API.

[1853] 4. The server analyzes the text data and uses Librosa to analyze the tone of voice.

[1854] 5. The server analyzes the user's facial expressions using OpenCV and Dlib.

[1855] 6. The emotion engine infers that the user is confused and feeds that information back to the generative AI.

[1856] 7. The generative AI adjusts the accuracy of its estimations to increase the likelihood of special fraud.

[1857] 8. The server notifies the user of the generative AI's inference results and issues a warning if necessary.

[1858] Prompt Sentence Examples

[1859] Convert the voice data when the user receives a call into text and analyze the user's emotional state. If the user is confused, adjust the estimation accuracy to increase the possibility of special fraud.

[1860] The above is a specific embodiment for carrying out the present invention.

[1861] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1862] Step 1:

[1863] The server accesses the user's email account and receives new emails. It extracts the body of the received email and saves it as text data. The input is the email data, and the output is the text of the email body. Specifically, it connects to the email server using the IMAP ...

Claims

1. A means to confirm the contents of emails or phone calls; Based on the prompt sentence that instructs the user to guess whether the confirmed content is a specialized fraud or not, A generative AI that estimates whether the confirmed content is a special fraud or not, A means for generating advice to the person and his / her family, including the possibility of fraud and countermeasures, based on the estimation by the generating system AI; A means for notifying the person and his / her family of the generated advice; A system including:

2. The system of claim 1 , wherein the generative AI infers whether or not the content is a specialized fraud by checking whether the content contains a typical pattern of specialized fraud.

3. The system of claim 1 , wherein the means for notifying the individual and his / her family of the advice notifies the individual and his / her family of the possibility of fraud and provides specific guidelines for action to protect against fraud.

Images