System

Abstract

A system is provided.SOLUTION: The system includes a means for analyzing the received e-mail and determining whether or not there is a suspicion of fraud, a means for acquiring and analyzing a call content and determining whether or not there is a suspicion of fraud, a means for reading a document with a camera and determining whether or not there is a suspicion of fraud, and a means for transmitting an alert to a user on the basis of the determination result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, the number of special fraud cases targeting the elderly has been increasing, and the resulting losses are extremely high. In order to combat these types of fraud, a means is needed to detect potential fraud early and warn users. However, there are few current security systems that are easy for the elderly to use, and the limited effectiveness of these systems in preventing fraud is a problem. For this reason, there is a need for a special fraud prevention system that is easy for the elderly to use. [Means for solving the problem]

[0005] The present invention provides a system that includes a means for analyzing received e-mails and determining whether they are suspected of fraud, a means for acquiring and analyzing the content of phone calls and determining whether they are suspected of fraud, a means for scanning documents with a camera and determining whether they are suspected of fraud, and a means for sending an alert to the user based on the determination results.The system also includes a means for registering and saving user information, and a means for monitoring calls in real time and sending an alert if the call contains content that is suspected of fraud.This reduces the risk of users, including the elderly, being involved in special frauds and ensures their safety.

[0006] "Received Email" means all readable messages that arrive at a user's email account via the Internet.

[0007] "Analysis" is the process of using pattern recognition and machine learning algorithms to examine data extracted from emails, calls, or documents to determine whether they are suspected of fraud.

[0008] "Suspected fraud" is a condition that warns that information or content is false or misleading and may result in financial loss or other adverse consequences.

[0009] "Determining" means determining whether the information is fraudulent based on the analysis results.

[0010] "Means" is a broad term that refers to a method, apparatus, or system used to accomplish a particular purpose.

[0011] "Call content" refers to all audio data of conversations conducted via landline or smartphone.

[0012] A "document" is any paper or mailing containing textual information recorded on paper or digital media.

[0013] "Camera scanning" refers to capturing image data of a document using an image capture device.

[0014] "Sending an alert" means notifying the user with a warning message by displaying or sounding it.

[0015] "Means for registering and saving user information" refers to the process of entering the user's personal information into the system and storing it in a database.

[0016] "Real-time monitoring" means continuous monitoring of call content as it occurs. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] As an embodiment of the present invention, various means constituting a special fraud prevention system (hereinafter referred to as the present system) will be described.

[0039] Registration and storage of user information

[0040] During the initial setup, the user uses the tablet device to enter the SSID and password to connect to their home Wi-Fi. After that, they enter personal information such as their name, age, and address, and the device sends this information to the server. The server receives the information and registers it in a database.

[0041] Email Analysis

[0042] The device monitors the user's email account and sends received emails to a server, which analyzes the email content and uses machine learning algorithms to determine whether the email is suspected of being fraudulent. If the analysis results indicate a suspected fraud, an alert is sent to the device.

[0043] Audio analysis of call content

[0044] When a user receives a suspected fraudulent call on their landline or smartphone, the device (smart speaker or smart earphone) records the call in real time and sends the audio data to a server. The server converts the audio data into text and determines whether it is suspected to be fraudulent. If it is, the device sends an alert to the user.

[0045] Mail analysis

[0046] The user takes a photo of a document, such as mail, with the tablet's camera. The device then sends the captured image data to a server. The server extracts text from the image and analyzes it to determine whether it is suspected of fraud. If it is suspected of fraud, the device sends an alert to the user.

[0047] Sending alerts

[0048] If fraud is suspected, the server generates an alert message and sends it to the device, which then displays the message on the screen of the smart speaker or tablet and, if necessary, notifies the smart earphones via voice notification.

[0049] Specific examples

[0050] Specific examples of mail analysis

[0051] 1. The user takes a photo of mail delivered to their home with their camera.

[0052] 2. The device sends the image data to the server.

[0053] 3. The server extracts text from the image and analyzes the text using machine learning algorithms.

[0054] 4. If it is determined that there is a suspicion of fraud, the device will send an alert to the user saying, "This mail is suspected to be fraudulent. Please be careful."

[0055] Specific examples of call analysis

[0056] 1. A user receives a suspected fraudulent call on their landline.

[0057] 2. The smart speaker device records the call and sends the data to the server.

[0058] 3. The server converts the speech into text and analyzes it.

[0059] 4. If fraud is suspected, the device will send an alert to the user saying, "This is a potential fraudulent call. Please hang up immediately."

[0060] In this way, the system can prevent the threat of special frauds and ensure safety for users, including the elderly.

[0061] The processing flow will be explained below.

[0062] Email Analysis

[0063] Step 1:

[0064] The user receives an email.

[0065] Step 2:

[0066] The terminal acquires the contents of the received email and sends them to the server.

[0067] Step 3:

[0068] The server analyzes the content of the email to determine whether it is suspected of being fraudulent.

[0069] Step 4:

[0070] The server sends the analysis results to the device.

[0071] Step 5:

[0072] If fraud is suspected, the device will display an alert or notify the user.

[0073] Audio analysis of call content

[0074] Step 1:

[0075] A user initiates a call on a desk phone or smartphone.

[0076] Step 2:

[0077] The device, a smart speaker or smart earphones, records the call in real time.

[0078] Step 3:

[0079] The device sends the recorded audio data to the server.

[0080] Step 4:

[0081] The server converts the audio data into text.

[0082] Step 5:

[0083] The server analyzes the converted text to determine whether it is suspected of fraud.

[0084] Step 6:

[0085] If fraud is suspected, the device will display an alert to the user or notify them audibly during the call.

[0086] Mail analysis

[0087] Step 1:

[0088] The user takes a photo of the mail that arrives at their home using the camera on their tablet device.

[0089] Step 2:

[0090] The terminal transmits the captured image data to the server.

[0091] Step 3:

[0092] The server extracts the text from the image.

[0093] Step 4:

[0094] The server analyzes the extracted text to determine whether it is suspected of fraud.

[0095] Step 5:

[0096] If fraud is suspected, the device will display an alert or notify the user.

[0097] account settings

[0098] Step 1:

[0099] The user opens the initial setup screen using the tablet device.

[0100] Step 2:

[0101] The user enters the Wi-Fi SSID and password.

[0102] Step 3:

[0103] The device attempts to connect to Wi-Fi and displays the results to the user.

[0104] Step 4:

[0105] The user enters personal information such as name, age, and address.

[0106] Step 5:

[0107] The terminal transmits the input information to the server.

[0108] Step 6:

[0109] The server receives the information and stores it in a database.

[0110] Step 7:

[0111] The server sends a confirmation message to the terminal that the save has been completed, and the terminal notifies the user.

[0112] Through these processing steps, the system is designed to ensure the safety of users and prevent special frauds from occurring.

[0113] Example 1

[0114] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0115] With the development of modern communication methods, frauds using email, telephone, and mail are on the rise. These frauds have become a problem that causes significant economic losses and psychological burdens to many people, including the elderly. Existing countermeasures have limitations, and more effective and comprehensive measures are needed.

[0116] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0117] In this invention, the server includes means for analyzing received emails and determining whether they are suspected of fraud using a machine learning algorithm, means for recording phone calls in real time, converting the audio data into text, and determining whether they are suspected of fraud, and means for scanning documents with a camera, extracting text from the image data, and determining whether they are suspected of fraud. This makes it possible to comprehensively detect fraudulent activities committed through multiple communication means and quickly warn users.

[0118] A "machine learning algorithm" is a computational method that allows a computer to learn patterns from data and make predictions or classifications.

[0119] "Voice data" refers to data that represents call content and voice information in digital format.

[0120] "Convert to text" is the process of converting non-text data, such as audio or images, into written information.

[0121] "Document" refers to information recorded in writing, including, for example, mail, letters, notices, etc.

[0122] A "camera" is a device that takes an image of an object and stores it as digital data.

[0123] "Image data" refers to data in which images taken with a camera or the like are stored in digital format.

[0124] "Text extraction" is the process of extracting textual information from image or audio data.

[0125] A "server" is a computer system that stores and processes data and provides services to clients over a network.

[0126] An "alert" is a notification that alerts or warns the user, and is provided as a voice or text message.

[0127] "User" refers to anyone who uses this system, and is primarily intended to prevent fraud.

[0128] As an embodiment of the present invention, various means of the special fraud prevention system (hereinafter referred to as the "system") will be specifically described. The system has the following main functions: registration and storage of user information, analysis of e-mails, analysis of voice of phone calls, analysis of mail, and sending of alerts.

[0129] Registration and storage of user information

[0130] The user performs initial setup using the tablet device. First, they enter the SSID and password to connect to their home Wi-Fi. Next, they enter personal information such as their name, age, and address. The entered information is sent from the device to a server, and the server registers the received information in a database. The software used in this process must support a basic user input form and SSL / TLS encrypted communication.

[0131] Email Analysis

[0132] The device monitors the user's email account and sends received emails to a server. The server uses machine learning algorithms to analyze the email content. This analysis process includes detecting fraud-related keywords and verifying sender information. If the analysis results indicate a suspected fraud, the server generates an alert and sends it to the device. The device then displays the alert to the user.

[0133] Examples:

[0134] When a user receives a fraudulent email from the email address "example@example.com," the device sends the contents of the email to the server. The server analyzes the email using a machine learning algorithm that includes the word "fraud." If the server determines that the email is suspected to be fraudulent, it displays a message on the device saying, "This email is suspected to be fraudulent. Do not open it."

[0135] Audio analysis of call content

[0136] When a user receives a suspected fraudulent call on their landline or smartphone, the device (smart speaker or smart earphone) records the call in real time. The audio data is sent to a server, which converts it into text. A machine learning algorithm is then used to determine whether the call is suspected to be fraudulent. If a fraudulent call is suspected, the device sends an alert to the user.

[0137] Examples:

[0138] When a user receives a "scam call pretending to be a relative" on their landline, the smart speaker records the call and sends the data to a server. The server converts the call, including keywords such as "money" and "transfer," into text and analyzes it. If the server determines that the call may be fraudulent, the device notifies the user, "This is a potential fraudulent call. Please hang up immediately."

[0139] Mail analysis

[0140] The user takes a photo of a document, such as mail, with the tablet's camera. The device then sends the captured image data to a server. The server then uses image processing technology to extract text from the image and analyzes the extracted text to determine whether there is any suspicion of fraud. If there is any suspicion of fraud, the device sends an alert to the user.

[0141] Examples:

[0142] A user takes a photo of fraudulent mail delivered to their home using the camera on their tablet device. The device sends the image data to a server, which then uses OCR technology to extract text such as "bank" and "transfer" from the document. If the analysis results indicate a suspicion of fraud, the device notifies the user, saying, "This mail is suspected to be fraudulent. Please be careful."

[0143] Sending alerts

[0144] If fraud is suspected, the server generates an alert message and sends it to the device, which then displays the message on the screen of the smart speaker or tablet and, if necessary, notifies the smart earphones via voice notification.

[0145] Examples:

[0146] Based on the result that "fraud is suspected," the server sends an alert message to the device stating "Warning: Possible fraud." The device displays the message on the tablet screen and uses smart earphones to notify the user by voice, "Possible fraud. Please be careful."

[0147] This system can prevent fraud committed through multiple communication methods against users, including the elderly. By using these methods, users are protected from the threat of fraud and can go about their daily lives with peace of mind.

[0148] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0149] Divide the program processing flow into processing steps

[0150] Registration and storage of user information

[0151] Step 1:

[0152] The user opens the Wi-Fi connection settings screen on the tablet device and enters the SSID and password to establish a connection.

[0153] Input: Wi-Fi SSID and password

[0154] Output: Wi-Fi connection establishment status

[0155] Step 2:

[0156] The user enters personal information (name, age, address, etc.) using a tablet device.

[0157] Input: Personal information such as name, age, address, etc.

[0158] Output: Temporarily save the entered personal information

[0159] Step 3:

[0160] The terminal encrypts the information entered and sends it to the server using the SSL / TLS protocol.

[0161] Input: User personal information

[0162] Output: Encrypted transmission of personal information

[0163] Step 4:

[0164] The server decrypts the received information and stores it in a secure database.

[0165] Input: Encrypted personal information

[0166] Output: Personal information stored in the database

[0167] Email Analysis

[0168] Step 1:

[0169] The device periodically checks the user's email account for new emails.

[0170] Input: Email account information

[0171] Output: New emails

[0172] Step 2:

[0173] The device receives new email and retrieves its contents (body, sender, attachments, etc.).

[0174] Input: New email

[0175] Output: Retrieved email content

[0176] Step 3:

[0177] The device encrypts the email content and sends it to the server.

[0178] Input: Email content

[0179] Output: Encrypted email content

[0180] Step 4:

[0181] The server decrypts the received email and analyzes it using machine learning algorithms.

[0182] Input: Encrypted email content

[0183] Output: Analysis results

[0184] Step 5:

[0185] If fraud is suspected, the server generates an alert message and sends it to the terminal.

[0186] Input: Analysis results

[0187] Output: Alert message

[0188] Step 6:

[0189] The device displays an alert to the user.

[0190] Input: Alert message

[0191] Output: User notification

[0192] Audio analysis of call content

[0193] Step 1:

[0194] Users receive suspicious fraudulent calls on their landline or smartphone.

[0195] Input:Incoming call

[0196] Output: Call Start

[0197] Step 2:

[0198] The device records the call in real time.

[0199] Input: Call content

[0200] Output: Recorded audio data

[0201] Step 3:

[0202] The device sends the recorded audio data to the server.

[0203] Input: Recorded audio data

[0204] Output: Transmitted audio data

[0205] Step 4:

[0206] The server converts the received voice data into text.

[0207] Input: Audio data

[0208] Output: The converted text

[0209] Step 5:

[0210] The server uses machine learning algorithms to analyze the text data.

[0211] Input: Translated text

[0212] Output: Analysis results

[0213] Step 6:

[0214] If fraud is suspected, the server generates an alert message and sends it to the terminal.

[0215] Input: Analysis results

[0216] Output: Alert message

[0217] Step 7:

[0218] The device notifies the user of the alert.

[0219] Input: Alert message

[0220] Output: User notification

[0221] Mail analysis

[0222] Step 1:

[0223] The user takes a photo of the mail item using the camera on the tablet device.

[0224] Input: Mail

[0225] Output: Image data

[0226] Step 2:

[0227] The terminal transmits the acquired image data to the server.

[0228] Input: Image data

[0229] Output: Image data sent

[0230] Step 3:

[0231] The server extracts the text from the image.

[0232] Input: Image data

[0233] Output: Extracted text

[0234] Step 4:

[0235] The server analyzes the extracted text using machine learning algorithms.

[0236] Input: Extracted text

[0237] Output: Analysis results

[0238] Step 5:

[0239] If fraud is suspected, the server generates an alert message and sends it to the terminal.

[0240] Input: Analysis results

[0241] Output: Alert message

[0242] Step 6:

[0243] The device notifies the user of the alert.

[0244] Input: Alert message

[0245] Output: User notification

[0246] Sending alerts

[0247] Step 1:

[0248] If fraud is suspected, the server generates an alert message.

[0249] Input: Analysis results

[0250] Output: Alert message

[0251] Step 2:

[0252] The server sends an alert message to the terminal.

[0253] Input: Alert message

[0254] Output: Alert message sent

[0255] Step 3:

[0256] The device will display an alert message on the screen of a smart speaker or tablet, and if necessary, will also notify the smart earphones via voice.

[0257] Input: The alert message sent

[0258] Output: User notification

[0259] This makes it possible to comprehensively detect fraudulent activities carried out through multiple communication means and quickly warn users.

[0260] (Application example 1)

[0261] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0262] In today's world, where users, including the elderly, are at increased risk of falling victim to special frauds, there is a need to prevent fraudulent acts via email, phone calls, and mail. However, it is difficult for users themselves to detect the signs of these frauds, so an efficient fraud prevention system is needed. Furthermore, conventional systems have had issues with the difficulty of real-time monitoring and highly accurate judgment.

[0263] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0264] In this invention, the server includes means for analyzing received emails and determining whether fraud is suspected, means for acquiring and analyzing phone call content and determining whether fraud is suspected, means for scanning documents with a photographing device and determining whether fraud is suspected, means for sending an alert to a user based on the determination result using a generative AI model, and means for generating a prompt message. This allows users to detect the risk of fraud via email, phone calls, or mail in real time and respond quickly.

[0265] "Means for analyzing received e-mails" refers to a function that analyzes the contents of e-mails received by the system and determines whether there is any suspicion of fraud.

[0266] "Means for acquiring and analyzing call content" refers to a function that acquires the content of a user's call, analyzes that content in real time, and determines whether there is any suspicion of fraud.

[0267] "Means for reading documents with a photographic device" refers to a function that allows a user to photograph a specific document with a photographic device such as a camera, and then determine whether or not there is suspicion of fraud based on the image data.

[0268] A "generative AI model" is an algorithmic model trained using machine learning and deep learning techniques that is used to analyze emails, calls, and documents to identify fraud.

[0269] "Means for generating prompts" refers to a function that uses an AI model to create and present appropriate warning messages or alerts to users based on specific conditions or judgment results.

[0270] "Means for sending alerts" refers to a function that sends an alert or message to a user's device to warn them if fraud is suspected.

[0271] As an embodiment of the present invention, a comprehensive system for protecting users, particularly the elderly, from special frauds is provided. The system includes means for automatically analyzing received emails, phone call contents, mail, etc., determining whether there is any suspected fraud, and sending an alert to the user.

[0272] Registration and storage of user information

[0273] During the initial setup, the user uses a tablet or smartphone to enter information (SSID and password) to connect to their home Wi-Fi. They also enter personal information such as their name, age, and address, and the device sends this information to the server. The server receives the information and registers it in a database.

[0274] Email Analysis

[0275] The email analysis method involves the device monitoring the user's email account and sending received emails to a server. The server then uses a generative AI model to analyze the email content and determine whether it is suspected of being fraudulent. If the analysis results indicate a suspicion of fraud, an alert is sent to the device.

[0276] Audio analysis of call content

[0277] When a user receives a suspected fraudulent call on their landline or smartphone, the device (smart speaker or smart earphone) records the call in real time and sends the audio data to a server. The server converts the audio data into text and uses a generative AI model to determine whether the call is suspected to be fraudulent. If fraud is suspected, the device sends the user an alert saying, "This is a potential fraudulent call. Please hang up immediately."

[0278] Mail analysis

[0279] A user takes a photo of a document, such as mail, with the camera on their tablet or smartphone. The device then sends the captured image data to a server. The server extracts text from the image and analyzes it using a generative AI model to determine whether it is suspected of fraud. If fraud is suspected, the device sends the user an alert saying, "This mail is suspected to be fraudulent. Please be careful."

[0280] Sending alerts

[0281] If the server determines that fraud is suspected, it generates a prompt message and sends it to the device, which then displays the message on the screen of the smart speaker or smartphone and, if necessary, sends a voice notification to the smart earphones.

[0282] Hardware and software used

[0283] Hardware: smartphones, tablets, smart speakers, smart earphones, cameras, microphones

[0284] Software: OCR software (e.g., tesseract-ocr), machine learning models (e.g., scikit-learn), speech recognition software (e.g., Google® Speech Recognition API), email processing libraries (e.g., python-email, smtplib), call monitoring software (e.g., Twilio API)

[0285] Prompt Sentence Examples

[0286] Use the following as an example prompt for analyzing suspected fraudulent emails:

[0287] "We have come across a potentially fraudulent email. The contents are as follows:

[0288] From: {sender}

[0289] Subject: {subject}

[0290] Contents: {body}

[0291] Please be careful as this appears to be a scam."

[0292] This will enable users to respond to fraudulent activity in real time, improving safety.

[0293] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0294] Step 1: Register and save user information

[0295] During the initial setup, a user uses a tablet or smartphone to enter information (SSID and password) to connect to their home Wi-Fi. They also enter personal information such as their name, age, and address, which the device then sends to the server. The server receives the information and registers it in a database. The input here is the user's Wi-Fi information and personal information, and the output is the user information stored in the database.

[0296] Step 2: Capture and parse emails

[0297] The device monitors the user's email account and sends received emails to a server. The server uses a generative AI model to analyze the email content and determine whether it is suspected of fraud. The input here is the content of the received email, and the output is a judgment result on whether it is suspected of fraud.

[0298] Step 3: Audio analysis of the call

[0299] When a user receives a suspected fraudulent call on their landline or smartphone, the device (smart speaker or smart earphone) records the call in real time and sends the audio data to a server. The server converts the audio data into text and uses a generative AI model to determine whether the call is suspected of fraud. The input here is the recorded call content, and the output is the converted text and a determination of whether the call is suspected of fraud.

[0300] Step 4: Image analysis of mail

[0301] A user takes a photo of a document, such as mail, using a tablet or smartphone camera. The device then sends the captured image data to a server. The server extracts text from the image and uses a generative AI model to analyze the text and determine whether it is suspected of fraud. The input here is the image of the mail, and the output is the extracted text and a determination of whether it is suspected of fraud.

[0302] Step 5: Sending an alert

[0303] If the server determines that fraud is suspected, it generates a prompt and an alert message and sends it to the device. The device then displays the warning message on the screen of the smart speaker or smartphone, and if necessary, notifies the smart earphones by voice. The input here is the judgment result regarding the suspected fraud, and the output is the generated prompt and alert message.

[0304] Through these steps, users can detect fraud risks in real time and respond quickly.

[0305] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0306] As an embodiment for carrying out the present invention, various means of a system (hereinafter referred to as an improved system) that combines a special fraud prevention system with an emotion engine will be described.

[0307] Registration and storage of user information

[0308] During the initial setup, the user uses the tablet device to enter the SSID and password to connect to their home Wi-Fi. After that, they enter personal information such as their name, age, and address, and the device sends this information to the server. The server receives the information and registers it in a database.

[0309] Email analysis and emotion recognition

[0310] The device monitors the user's email account and sends received emails to a server. The server analyzes the email content and determines whether it is suspected of being fraudulent. It also recognizes the user's emotions based on the email content. If the analysis results indicate a suspicion of fraud, it sends an alert to the device.

[0311] Voice analysis and emotion recognition of call content

[0312] When a user receives a suspected fraudulent call on their landline or smartphone, the smart speaker or earphone device records the call in real time and sends the audio data to a server. The server converts the audio data into text and then analyzes the text to determine whether the call is suspected of fraud. The system also recognizes the user's emotions from the call content and sends an alert if there is a change in emotion.

[0313] Mail analysis and emotion recognition

[0314] The user takes a photo of a document, such as mail, using the tablet's camera. The device then sends the captured image data to a server. The server extracts text from the image and analyzes it to determine whether it is suspected of fraud. In addition, the system recognizes the user's emotions based on the document's content. If it determines that the document is suspected of fraud, the device sends an alert to the user.

[0315] Sending alerts

[0316] If fraud is suspected, the server generates an alert message and sends it to the device. The device then displays the message on the screen of the smart speaker or tablet, and if necessary, sends a voice notification to the smart earphones. The emotion engine also notifies the user of the analysis results and provides appropriate advice based on the user's psychological state.

[0317] Specific examples

[0318] Specific examples of mail analysis

[0319] 1. The user takes a photo of mail delivered to their home with their camera.

[0320] 2. The device sends the image data to the server.

[0321] 3. The server extracts text from the image and analyzes the text using machine learning algorithms.

[0322] 4. The server further recognizes the user's emotions based on the document content.

[0323] 5. If it determines that there is a suspicion of fraud, the device will send the user an alert saying, "This mail is suspected to be fraudulent. Please be careful," and will also send an emotional alert saying, "This mail conveys a sense of anxiety. Please ignore it."

[0324] Specific examples of call analysis

[0325] 1. A user receives a suspected fraudulent call on their landline.

[0326] 2. The smart speaker device records the call and sends the data to the server.

[0327] 3. The server converts the speech into text and analyzes it.

[0328] 4. The server recognizes the user's emotions from the content of the call and sends an alert if there is a change in emotions.

[0329] 5. If fraud is suspected, the device will send the user an alert saying, "This is a potential fraudulent call. Please hang up immediately," and will also provide an emotional alert saying, "You are feeling surprised and anxious. Please hang up."

[0330] As a result, the improved system can prevent the threat of special fraud and ensure safety for users, including the elderly. In addition, it can recognize the user's emotional state in real time and provide psychological support.

[0331] The processing flow will be explained below.

[0332] Email analysis and emotion recognition

[0333] Step 1:

[0334] The user receives an email.

[0335] Step 2:

[0336] The terminal acquires the contents of the received email and sends them to the server.

[0337] Step 3:

[0338] The server analyzes the content of the email to determine whether it is suspected of being fraudulent.

[0339] Step 4:

[0340] The server recognizes the user's emotions based on the email content.

[0341] Step 5:

[0342] The server sends the analysis results and emotion recognition results to the device.

[0343] Step 6:

[0344] If fraud is suspected, the device will display or notify the user of an alert and an emotion alert.

[0345] Voice analysis and emotion recognition of call content

[0346] Step 1:

[0347] A user initiates a call on a desk phone or smartphone.

[0348] Step 2:

[0349] The device, a smart speaker or smart earphones, records the call in real time.

[0350] Step 3:

[0351] The device sends the recorded audio data to the server.

[0352] Step 4:

[0353] The server converts the audio data into text.

[0354] Step 5:

[0355] The server analyzes the converted text to determine whether it is suspected of fraud.

[0356] Step 6:

[0357] The server recognizes the user's emotions based on the contents of the call.

[0358] Step 7:

[0359] The server sends the analysis results and emotion recognition results to the device.

[0360] Step 8:

[0361] If fraud is suspected, the device will display an alert and emotion alert to the user or notify them audibly during the call.

[0362] Mail analysis and emotion recognition

[0363] Step 1:

[0364] The user takes a photo of the mail that arrives at their home using the camera on their tablet device.

[0365] Step 2:

[0366] The terminal transmits the captured image data to the server.

[0367] Step 3:

[0368] The server extracts the text from the image.

[0369] Step 4:

[0370] The server analyzes the extracted text to determine whether it is suspected of fraud.

[0371] Step 5:

[0372] The server recognizes the user's emotions based on the document content.

[0373] Step 6:

[0374] The server sends the analysis results and emotion recognition results to the device.

[0375] Step 7:

[0376] If fraud is suspected, the device will display or notify the user of an alert and an emotion alert.

[0377] account settings

[0378] Step 1:

[0379] The user opens the initial setup screen using the tablet device.

[0380] Step 2:

[0381] The user enters the Wi-Fi SSID and password.

[0382] Step 3:

[0383] The device attempts to connect to Wi-Fi and displays the results to the user.

[0384] Step 4:

[0385] The user enters personal information such as name, age, and address.

[0386] Step 5:

[0387] The terminal transmits the input information to the server.

[0388] Step 6:

[0389] The server receives the information and stores it in a database.

[0390] Step 7:

[0391] The server sends a confirmation message to the terminal that the save has been completed, and the terminal notifies the user.

[0392] Through these processing steps, the improved system can ensure the safety of users and prevent them from committing fraud. Furthermore, by utilizing the emotion engine, it is possible to determine the user's emotional state in real time and provide psychological support.

[0393] Example 2

[0394] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0395] In modern society, special frauds are becoming increasingly sophisticated and their methods are diversifying. Fraud targeting the elderly in particular has become a major social problem, and conventional fraud prevention systems are often unable to keep up with new methods. Furthermore, the lack of appropriate countermeasures and support that take into account the user's emotional state makes it difficult to prevent damage before it occurs. To solve this problem, it is necessary to provide a system that can effectively analyze communication methods (email, phone calls, documents) that are suspected of being fraudulent, and that can recognize and respond to the user's emotional state in real time.

[0396] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0397] In this invention, the server includes means for analyzing received electronic messages and determining whether fraud is suspected, means for acquiring and analyzing the contents of phone calls and determining whether fraud is suspected, means for photographing and reading documents and determining whether fraud is suspected, means for analyzing the emotional state of the user, and means for sending an alert to the user based on the determination result and the emotional state. This makes it possible to deal with a variety of fraudulent methods, provide appropriate support that takes into account the psychological state of the user, and prevent fraud damage before it occurs.

[0398] "Means for analyzing received electronic messages to determine whether they are suspected of fraud" refers to a combination of software and hardware that processes the content of emails and messages received by a user and automatically detects possible fraudulent activity.

[0399] "Means for acquiring and analyzing call content to determine whether fraud is suspected" refers to a combination of software and hardware that records the user's call content and analyzes the audio data to identify possible fraud.

[0400] "Means for photographing and reading documents to determine whether they are suspected of fraud" refers to a combination of software and hardware that converts the contents of a document photographed by a user with a camera into text data using technologies such as optical character recognition, and then analyzes the text data to determine whether it is suspected of fraud.

[0401] The "means for analyzing a user's emotional state" is a combination of software and hardware for analyzing a user's emotional state based on the contents of electronic messages, conversations and documents, and evaluating the results.

[0402] The "means for sending an alert to the user based on the judgment result and emotional state" refers to a combination of software and hardware that, when a suspected fraud is judged to be present, generates a warning message that takes into account the judgment result and the user's emotional state and notifies the user.

[0403] "Means for registering and storing user information" refers to a combination of software and hardware for registering users' personal information in a database and using or referencing it when necessary.

[0404] "Means for monitoring a user's voice communications in real time and sending an alert if the communications contain suspected fraudulent content" refers to a combination of software and hardware that monitors a user's calls in real time and immediately issues an alert if suspected fraudulent content is detected.

[0405] As an embodiment of the present invention, the detailed configuration and operation of a system (hereinafter referred to as "this system") that combines a special fraud prevention system with an emotion recognition engine will be described.

[0406] Registration and storage of user information

[0407] The user uses the tablet device for initial setup and enters the SSID and password to connect to their home Wi-Fi. Next, they enter personal information such as their name, age, and address into the device. This information is sent from the device to the server. The server registers the received information in a database, which allows the user's basic information to be safely stored.

[0408] Analysis of electronic messages and emotion recognition

[0409] The device periodically monitors the user's email account and receives new emails. The received emails are immediately sent to the server. The server uses a text analysis engine to analyze the email content and determine whether it is suspected of being fraudulent. The server also uses an emotion recognition engine to extract the user's emotions from the email content. If it determines that the email is suspected of being fraudulent, it sends an alert message to the device and displays a warning to the user.

[0410] Voice analysis and emotion recognition of call content

[0411] When a user receives a suspected fraudulent call on their landline or smartphone, the smart speaker or earphones that are part of the device record the call in real time. The recorded data is then immediately sent to a server. The server uses a speech recognition engine to convert the voice data into text and analyzes the text to determine whether the call is suspected of fraud. The server also recognizes the user's emotions from the call content and sends an alert if a change in emotion is detected.

[0412] Document Analysis and Emotion Recognition

[0413] The user takes a photo of a document, such as mail, with the tablet's camera. The device then sends the captured image data to the server. The server uses an image analysis engine to extract text from the image and analyzes that text to determine whether it is suspected of fraud. The server then recognizes the user's emotions based on the document's content. If it determines that the document is suspected of fraud, the device sends an alert to the user.

[0414] Sending alerts

[0415] If a suspected fraud attempt is detected, the server generates an alert message and sends it to the device, which then displays the message on the screen of the smart speaker or tablet and, if necessary, also notifies the smart earphones via voice notification, allowing the user to immediately become aware of the possibility of fraud.

[0416] Specific examples

[0417] Specific examples of mail analysis

[0418] 1. The user takes a photo of the mail that arrives at their home with their camera.

[0419] 2. The device sends the image data to the server.

[0420] 3. The server extracts text from the image and analyzes the text using machine learning algorithms.

[0421] 4. The server further recognizes the user's emotions based on the document content.

[0422] 5. If it determines that there is a suspicion of fraud, the device will send the user an alert saying, "This mail is suspected to be fraudulent. Please be careful," and will also send an emotional alert saying, "This mail conveys a sense of anxiety. Please ignore it."

[0423] Specific examples of call analysis

[0424] 1. A user receives a suspected fraudulent call on their landline.

[0425] 2. The smart speaker device records the call and sends the data to the server.

[0426] 3. The server converts the speech into text and analyzes it.

[0427] 4. The server recognizes the user's emotions from the content of the call and sends an alert if there is a change in emotion.

[0428] 5. If fraud is suspected, the device will send the user an alert saying, "This is a potential fraudulent call. Please hang up immediately," and will also provide an emotional alert saying, "You are feeling surprised and anxious. Please hang up."

[0429] In this way, this system can prevent the threat of special fraud and ensure the safety of users, including the elderly. In addition, it can recognize the user's emotional state in real time and provide psychological support.

[0430] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0431] Registration and storage of user information

[0432] Processing Steps

[0433] Step 1:

[0434] The user opens the settings screen on the tablet device and enters the SSID and password to connect to their home Wi-Fi.

[0435] Input: SSID, password

[0436] Output: Wi-Fi connection complete message

[0437] Specific operation: The user enters the SSID and password in the specified fields and presses the "Connect" button. The device receives this and attempts to connect to Wi-Fi. If the connection is successful, a connection completion message is displayed.

[0438] Step 2:

[0439] The user enters personal information such as name, age, and address into the terminal.

[0440] Input: Personal information such as name, age, address, etc.

[0441] Output: Input confirmation screen

[0442] Specific operation: The user enters personal information into each field and presses the "Submit" button. The device retrieves this information and displays a confirmation screen.

[0443] Step 3:

[0444] The terminal transmits the entered personal information to the server.

[0445] Input: Personal information entered on the device

[0446] Output: Message that data has been sent to the server

[0447] Specific operation: The terminal packetizes the entered personal information and sends it to the server using a secure communication protocol (e.g., HTTPS).

[0448] Step 4:

[0449] The server registers the received personal information in a database.

[0450] Input: Personal information sent from the device

[0451] Output: Database registration completion message

[0452] Specific operation: The server analyzes the received data and stores it in the database appropriately. Once the data has been stored, it returns a registration completion message to the device.

[0453] Analysis of electronic messages and emotion recognition

[0454] Processing Steps

[0455] Step 1:

[0456] The device periodically monitors the user's email account and receives new emails.

[0457] Input: Email login details

[0458] Output: New email information

[0459] Specific operation: The device sends a query to the mail server at the specified interval to check for new mail. If new mail is found, the device retrieves its contents.

[0460] Step 2:

[0461] The device sends the contents of the new email to the server.

[0462] Input: New email content

[0463] Output: Message that email has been sent to the server

[0464] Specific operation: The device obtains the body and header information of the new email and sends it to the server.

[0465] Step 3:

[0466] The server uses a text analysis engine to analyze the email content and determine whether it is suspected of being fraudulent.

[0467] Input: Email text data

[0468] Output: Fraud detection result

[0469] What it does: The server uses a text analysis engine (e.g., an NLP model) to analyze the content of the email and determine whether it is fraudulent. The result is stored as a flag.

[0470] Step 4:

[0471] The server uses an emotion recognition engine to extract the user's emotion from the content of the email.

[0472] Input: Email text data

[0473] Output: Emotion recognition result

[0474] Specific operation: The server uses an emotion recognition engine (e.g., emotion analysis API) to extract the user's emotion from the email body and saves the result as a flag.

[0475] Step 5:

[0476] If fraud is suspected, an alert message will be sent to the terminal and a warning will be displayed to the user.

[0477] Input: Fraud detection result, emotion recognition result

[0478] Output: Alert message screen

[0479] Specific operation: If the fraud flag is set, the server generates an alert message and sends it to the terminal. The terminal receives it and displays the warning message to the user.

[0480] Voice analysis and emotion recognition of call content

[0481] Processing Steps

[0482] Step 1:

[0483] Users receive suspicious fraudulent calls on their landline or smartphone.

[0484] Input: Incoming call notification

[0485] Output: Call start signal

[0486] Specific operation: The user receives an incoming call and answers the phone. The device detects this and transitions to a call state.

[0487] Step 2:

[0488] Smart speakers and smart earphones that are part of the device record the contents of calls in real time.

[0489] Input: Call audio data

[0490] Output: Recording data

[0491] Specific operation: The terminal starts recording as soon as the call starts and stores the audio data in a buffer.

[0492] Step 3:

[0493] The recording data is immediately sent to the server.

[0494] Input: Recording data

[0495] Output: Message that data has been sent to the server

[0496] What happens: The device sends the recording data to the server using a secure protocol (e.g., HTTPS).

[0497] Step 4:

[0498] The server uses a speech recognition engine to convert the voice data into text.

[0499] Input: Audio data

[0500] Output: Text data

[0501] Specific operation: The server uses a speech recognition engine (e.g., Speech-to-Text API) to convert the voice data into text.

[0502] Step 5:

[0503] The server analyzes the text to determine if it is suspected of fraud.

[0504] Input: Converted text data

[0505] Output: Fraud detection result

[0506] What it does: The server uses a text analysis engine to determine the likelihood of fraud, and stores the result as a flag.

[0507] Step 6:

[0508] The server recognizes the user's emotions from the content of the call and sends an alert if there is a change in emotion.

[0509] Input: Text data

[0510] Output: Emotion recognition result

[0511] Specific operation: The server uses an emotion recognition engine to extract the user's emotion from the text data and saves the result as a flag.

[0512] Step 7:

[0513] If fraud is suspected, an alert message will be sent to the terminal and a warning will be displayed to the user.

[0514] Input: Fraud detection result, emotion recognition result

[0515] Output: Alert message screen

[0516] Specific operation: If the fraud flag is set, the server generates an alert message and sends it to the terminal. The terminal receives it and displays the warning message to the user.

[0517] Document Analysis and Emotion Recognition

[0518] Processing Steps

[0519] Step 1:

[0520] The user takes a photo of the mail item using the camera on the tablet device.

[0521] Input: Image data of mail

[0522] Output: Shooting completion message

[0523] Specific operation: The user takes a photo of the mail using the camera function of the tablet device. The image data is stored on the device.

[0524] Step 2:

[0525] The terminal transmits the image data to the server.

[0526] Input: Image data of mail

[0527] Output: Message that data has been sent to the server

[0528] Specific operation: The device sends the acquired image data to the server using a secure protocol (e.g., HTTPS).

[0529] Step 3:

[0530] The server uses an image analysis engine to extract text from the image.

[0531] Input: Image data

[0532] Output: Extracted text data

[0533] Specific operation: The server uses an image analysis engine (e.g., OCR API) to extract text data from the image.

[0534] Step 4:

[0535] The server analyzes the text to determine if it is suspected of fraud.

[0536] Input: Extracted text data

[0537] Output: Fraud detection result

[0538] What it does: The server uses a text analysis engine to determine the likelihood of fraud, and stores the result as a flag.

[0539] Step 5:

[0540] The server recognizes the user's emotions based on the document content.

[0541] Input: Extracted text data

[0542] Output: Emotion recognition result

[0543] Specific operation: The server uses an emotion recognition engine to extract the user's emotion from the document content, and the result is saved as a flag.

[0544] Step 6:

[0545] If fraud is suspected, an alert message will be sent to the terminal and a warning will be displayed to the user.

[0546] Input: Fraud detection result, emotion recognition result

[0547] Output: Alert message screen

[0548] Specific operation: If the fraud flag is set, the server generates an alert message and sends it to the terminal. The terminal receives it and displays a warning message to the user.

[0549] Sending alerts

[0550] Processing Steps

[0551] Step 1:

[0552] The server detects suspected fraud.

[0553] Input: Fraud detection result, emotion recognition result

[0554] Output: Alert message

[0555] Specific operation: The server checks the fraud determination flag and emotion recognition flag and generates an alert message if necessary.

[0556] Step 2:

[0557] The server generates an alert message and sends it to the terminal.

[0558] Input: Fraud detection result, emotion recognition result

[0559] Output: The generated alert message

[0560] Specific operation: The server generates an alert message and sends it to the terminal in an appropriate format.

[0561] Step 3:

[0562] The device displays or notifies you of an alert message.

[0563] Input: The generated alert message

[0564] Output: The displayed warning message screen

[0565] Specific operation: When the terminal receives an alert message, it displays a visual or audio warning message to the user.

[0566] Through the above processing steps, the system can provide high detection accuracy and rapid response to a variety of fraudulent methods, while also providing appropriate support based on the user's emotional state.

[0567] (Application example 2)

[0568] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0569] In recent years, there has been an increase in special frauds targeting the elderly, and many of these are committed via email, telephone, or mail. This has led to a growing need for systems that can not only detect fraud but also monitor users' emotions in real time and provide appropriate responses based on their psychological state. While conventional systems specialize in detecting fraud, they do not provide support that takes users' emotions into consideration, resulting in a lack of psychological support.

[0570] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0571] In this invention, the server includes means for analyzing received electronic documents and determining whether fraud is suspected, means for acquiring and analyzing the contents of phone calls and determining whether fraud is suspected, means for scanning documents with an imaging device and determining whether fraud is suspected, means for recognizing the user's emotions using an emotion recognition engine when determining whether fraud is suspected, and providing appropriate advice on how to respond if there is a possibility of fraud, and means for sending a warning to the user based on the determination result. This makes it possible not only to respond immediately to information that is suspected of being fraud, but also to provide psychological support that takes into account the user's emotional state.

[0572] An "electronic document" is a document stored in digital format, including email, PDF, Word files, etc.

[0573] "Call content" refers to information exchanged through voice communication, such as the content of a conversation conducted via telephone or internet telephone.

[0574] "Photographing device" refers to equipment for taking still or moving images, and includes devices with camera functions.

[0575] An "emotion recognition engine" is a combination of software and hardware for identifying and analyzing human emotional states.

[0576] A "warning" is a message or notification that warns the user of a caution or danger based on a specific condition.

[0577] "User Information" refers to personal data relating to users of the system, including name, age, address, contact details, etc.

[0578] "Analysis" refers to the process of examining a set of data or information in detail to understand its content, trends, and relationships.

[0579] "Judgment" is the act of determining an outcome based on collected data and information according to specific conditions and criteria.

[0580] System Overview

[0581] This system is a security system that combines fraud prevention and user emotion recognition. The system analyzes the contents of received electronic documents, phone calls, and documents captured using a camera, and issues a warning if fraud is suspected. Furthermore, the system uses an emotion recognition engine to analyze the user's emotional state and provide appropriate responses.

[0582] Hardware and software used

[0583] Hardware:

[0584] Smartphone

[0585] camera

[0586] software:

[0587] Server side: Django

[0588] Emotion recognition API: Azure(R) Cognitive Services Emotion API

[0589] NLP libraries: spaCy, nltk

[0590] Deep Learning: TENSORFLOW (registered trademark), PyTorch

[0591] Speech Recognition: Google Speech-to-Text API

[0592] OCR library: Tesseract

[0593] Program processing

[0594] 1. Electronic Document Analysis:

[0595] The server periodically retrieves emails received via the API and performs text analysis using an NLP library (spaCy or nltk).

[0596] It analyzes text structure and keywords, and if there is a possibility of fraud, it uses the Azure Cognitive Services Emotion API to analyze the user's emotions.

[0597] If fraud is suspected, a warning message will be generated and sent to the user's smartphone.

[0598] 2. Call analysis:

[0599] It monitors smartphone calls in real time and converts audio data into text using the Google Speech-to-Text API.

[0600] Use NLP libraries to analyze text and determine potential fraud.

[0601] The Azure Cognitive Services Emotion API analyzes the user's emotions, and if anxiety or surprise is detected, a warning message and emotion-based advice is sent to the user.

[0602] 3. Document Analysis:

[0603] A photograph of a document such as mail is taken using a photographing device (smartphone camera), and the image data is sent to the server.

[0604] The server uses an OCR library (Tesseract) to extract text from image data and performs text analysis using an NLP library.

[0605] If fraud is suspected, emotion recognition is performed, a message advising appropriate action is generated, and a warning is sent to the user's smartphone.

[0606] Specific examples

[0607] Specific examples of email analysis

[0608] Process flow: Retrieve email from SMTP server → Analyze text using spaCy → Analyze emotions using Azure Cognitive Services Emotion API → Warn user if fraud is suspected

[0609] Example alerts: "This email may be fraudulent. Please be cautious.", "This email makes me feel uneasy. Please ignore it."

[0610] Prompt Sentence Examples

[0611] "Please analyze the content of the email below to determine the likelihood of a scam and the associated emotions.

[0612] The email reads: "Dear customer, you have been selected for our special campaign. Please update your account information to receive your prize."

[0613] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0614] Step 1: Obtaining electronic documents

[0615] Description: The server periodically retrieves the user's email via API.

[0616] Input: User's email account information

[0617] Output: Retrieved email data

[0618] What happens: The server fetches emails through the SMTP server's API and retrieves new emails in the inbox.

[0619] Step 2: Text analysis

[0620] Description: The server performs text analysis on the body of the retrieved email.

[0621] Input: Email data

[0622] Output: Analysis results (suspected fraud)

[0623] What it does: The server uses an NLP library (e.g. spaCy) to tokenize the email body, extract keywords, and perform contextual analysis to assess the likelihood of fraud.

[0624] Step 3: Sentiment Analysis

[0625] Description: The server analyzes the user's emotional state based on the content of the email.

[0626] Input: Email data, NLP analysis results

[0627] Output: Emotion analysis results

[0628] How it works: The server uses the Azure Cognitive Services Emotion API to estimate the emotions the user may be feeling from the content of the email.

[0629] Step 4: Generate warning messages

[0630] Description: If fraud is suspected, the server generates a warning message.

[0631] Input: NLP analysis results, sentiment analysis results

[0632] Output: Warning message

[0633] Specific behavior: The server generates a warning message based on the analysis results, for example, "This email may be fraudulent. Please be careful."

[0634] Step 5: Sending an alert

[0635] Description: The server sends the generated warning message to the user's terminal.

[0636] Input: warning message

[0637] Output: A warning displayed on the user's terminal

[0638] Specific operation: The server sends a warning message to the user's smartphone via push notification.

[0639] Step 6: Capture the call

[0640] Description: The device receives the user's call content in real time.

[0641] Input: Call audio data

[0642] Output: Recorded audio data

[0643] How it works: The smartphone uses its built-in microphone to record the conversation and sends it to the server as audio data.

[0644] Step 7: Speech to Text

[0645] Description: The server converts the call audio data into text.

[0646] Input: Captured audio data

[0647] Output: Text data

[0648] Specific operation: The server uses the Google Speech-to-Text API to convert the captured voice data into text.

[0649] Step 8: Call Analysis

[0650] Description: The server analyzes the text of the call to determine if it is suspected of fraud.

[0651] Input: Text data

[0652] Output: Analysis results (suspected fraud)

[0653] What it does: The server uses an NLP library to tokenize the call, extract keywords and perform contextual analysis to assess the likelihood of fraud.

[0654] Step 9: Sentiment Analysis

[0655] Description: The server analyzes the user's emotional state from the content of the call.

[0656] Input: Text data, NLP analysis results

[0657] Output: Emotion analysis results

[0658] Specific operation: The server uses the Azure Cognitive Services Emotion API to estimate the user's emotions from the content of the call.

[0659] Step 10: Generate warning messages

[0660] Description: If fraud is suspected or there is a change in sentiment, the server generates a warning message.

[0661] Input: NLP analysis results, sentiment analysis results

[0662] Output: Warning message

[0663] Specific behavior: Based on the analysis results, the server generates a warning message such as "This call may be fraudulent. Please hang up immediately."

[0664] Step 11: Sending an alert

[0665] Description: The server sends the generated warning message to the user's terminal.

[0666] Input: warning message

[0667] Output: A warning displayed on the user's terminal

[0668] Specific operation: The server sends a warning message to the user's smartphone via push notification.

[0669] Step 12: Photograph the document

[0670] Description: A user takes a picture of a document, such as mail, with a camera.

[0671] Input: Image data

[0672] Output: Photographed document image

[0673] Specific action: The user uses the smartphone camera to take a picture of the mail.

[0674] Step 13: Extracting text from images

[0675] Description: The server extracts text from the captured image.

[0676] Input: Image data

[0677] Output: Extracted text data

[0678] Specific operation: The server uses an OCR library (Tesseract) to extract text from the image.

[0679] Step 14: Document Content Analysis

[0680] Description: The server analyzes the text data to determine if it is suspected of fraud.

[0681] Input: Text data

[0682] Output: Analysis results (suspected fraud)

[0683] How it works: The server uses an NLP library to tokenize the document content, extract keywords and perform contextual analysis to assess the likelihood of fraud.

[0684] Step 15: Sentiment Analysis

[0685] Description: The server analyzes the user's emotional state from the document content.

[0686] Input: Text data, NLP analysis results

[0687] Output: Emotion analysis results

[0688] Specific operation: The server uses the Azure Cognitive Services Emotion API to estimate the user's emotions from the document content.

[0689] Step 16: Generate warning messages

[0690] Description: If fraud is suspected, the server generates a warning message.

[0691] Input: NLP analysis results, sentiment analysis results

[0692] Output: Warning message

[0693] Specific operation: Based on the analysis results, the server creates a message such as "This mail is suspected to be fraudulent. Please be careful."

[0694] Step 17: Sending Alerts

[0695] Description: The server sends the generated warning message to the user's terminal.

[0696] Input: warning message

[0697] Output: A warning displayed on the user's terminal

[0698] Specific operation: The server sends a warning message to the user's smartphone via push notification.

[0699] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0700] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0701] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0702] [Second embodiment]

[0703] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0704] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0705] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0706] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0707] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0708] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0709] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0710] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0711] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0712] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0713] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0714] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0715] As an embodiment of the present invention, various means constituting a special fraud prevention system (hereinafter referred to as the present system) will be described.

[0716] Registration and storage of user information

[0717] During the initial setup, the user uses the tablet device to enter the SSID and password to connect to their home Wi-Fi. After that, they enter personal information such as their name, age, and address, and the device sends this information to the server. The server receives the information and registers it in a database.

[0718] Email Analysis

[0719] The device monitors the user's email account and sends received emails to a server, which analyzes the email content and uses machine learning algorithms to determine whether the email is suspected of being fraudulent. If the analysis results indicate a suspected fraud, an alert is sent to the device.

[0720] Audio analysis of call content

[0721] When a user receives a suspected fraudulent call on their landline or smartphone, the device (smart speaker or smart earphone) records the call in real time and sends the audio data to a server. The server converts the audio data into text and determines whether it is suspected to be fraudulent. If it is, the device sends an alert to the user.

[0722] Mail analysis

[0723] The user takes a photo of a document, such as mail, with the tablet's camera. The device then sends the captured image data to a server. The server extracts text from the image and analyzes it to determine whether it is suspected of fraud. If it is suspected of fraud, the device sends an alert to the user.

[0724] Sending alerts

[0725] If fraud is suspected, the server generates an alert message and sends it to the device, which then displays the message on the screen of the smart speaker or tablet and, if necessary, notifies the smart earphones via voice notification.

[0726] Specific examples

[0727] Specific examples of mail analysis

[0728] 1. The user takes a photo of mail delivered to their home with their camera.

[0729] 2. The device sends the image data to the server.

[0730] 3. The server extracts text from the image and analyzes the text using machine learning algorithms.

[0731] 4. If it is determined that there is a suspicion of fraud, the device will send an alert to the user saying, "This mail is suspected to be fraudulent. Please be careful."

[0732] Specific examples of call analysis

[0733] 1. A user receives a suspected fraudulent call on their landline.

[0734] 2. The smart speaker device records the call and sends the data to the server.

[0735] 3. The server converts the speech into text and analyzes it.

[0736] 4. If fraud is suspected, the device will send an alert to the user saying, "This is a potential fraudulent call. Please hang up immediately."

[0737] In this way, the system can prevent the threat of special frauds and ensure safety for users, including the elderly.

[0738] The processing flow will be explained below.

[0739] Email Analysis

[0740] Step 1:

[0741] The user receives an email.

[0742] Step 2:

[0743] The terminal acquires the contents of the received email and sends them to the server.

[0744] Step 3:

[0745] The server analyzes the content of the email to determine whether it is suspected of being fraudulent.

[0746] Step 4:

[0747] The server sends the analysis results to the device.

[0748] Step 5:

[0749] If fraud is suspected, the device will display an alert or notify the user.

[0750] Audio analysis of call content

[0751] Step 1:

[0752] A user initiates a call on a desk phone or smartphone.

[0753] Step 2:

[0754] The device, a smart speaker or smart earphones, records the call in real time.

[0755] Step 3:

[0756] The device sends the recorded audio data to the server.

[0757] Step 4:

[0758] The server converts the audio data into text.

[0759] Step 5:

[0760] The server analyzes the converted text to determine whether it is suspected of fraud.

[0761] Step 6:

[0762] If fraud is suspected, the device will display an alert to the user or notify them audibly during the call.

[0763] Mail analysis

[0764] Step 1:

[0765] The user takes a photo of the mail that arrives at their home using the camera on their tablet device.

[0766] Step 2:

[0767] The terminal transmits the captured image data to the server.

[0768] Step 3:

[0769] The server extracts the text from the image.

[0770] Step 4:

[0771] The server analyzes the extracted text to determine whether it is suspected of fraud.

[0772] Step 5:

[0773] If fraud is suspected, the device will display an alert or notify the user.

[0774] account settings

[0775] Step 1:

[0776] The user opens the initial setup screen using the tablet device.

[0777] Step 2:

[0778] The user enters the Wi-Fi SSID and password.

[0779] Step 3:

[0780] The device attempts to connect to Wi-Fi and displays the results to the user.

[0781] Step 4:

[0782] The user enters personal information such as name, age, and address.

[0783] Step 5:

[0784] The terminal transmits the input information to the server.

[0785] Step 6:

[0786] The server receives the information and stores it in a database.

[0787] Step 7:

[0788] The server sends a confirmation message to the terminal that the save has been completed, and the terminal notifies the user.

[0789] Through these processing steps, the system is designed to ensure the safety of users and prevent special frauds from occurring.

[0790] Example 1

[0791] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0792] With the development of modern communication methods, frauds using email, telephone, and mail are on the rise. These frauds have become a problem that causes significant economic losses and psychological burdens to many people, including the elderly. Existing countermeasures have limitations, and more effective and comprehensive measures are needed.

[0793] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0794] In this invention, the server includes means for analyzing received emails and determining whether they are suspected of fraud using a machine learning algorithm, means for recording phone calls in real time, converting the audio data into text, and determining whether they are suspected of fraud, and means for scanning documents with a camera, extracting text from the image data, and determining whether they are suspected of fraud. This makes it possible to comprehensively detect fraudulent activities committed through multiple communication means and quickly warn users.

[0795] A "machine learning algorithm" is a computational method that allows a computer to learn patterns from data and make predictions or classifications.

[0796] "Voice data" refers to data that represents call content and voice information in digital format.

[0797] "Convert to text" is the process of converting non-text data, such as audio or images, into written information.

[0798] "Document" refers to information recorded in writing, including, for example, mail, letters, notices, etc.

[0799] A "camera" is a device that takes an image of an object and stores it as digital data.

[0800] "Image data" refers to data in which images taken with a camera or the like are stored in digital format.

[0801] "Text extraction" is the process of extracting textual information from image or audio data.

[0802] A "server" is a computer system that stores and processes data and provides services to clients over a network.

[0803] An "alert" is a notification that alerts or warns the user, and is provided as a voice or text message.

[0804] "User" refers to anyone who uses this system, and is primarily intended to prevent fraud.

[0805] As an embodiment of the present invention, various means of the special fraud prevention system (hereinafter referred to as the "system") will be specifically described. The system has the following main functions: registration and storage of user information, analysis of e-mails, analysis of voice of phone calls, analysis of mail, and sending of alerts.

[0806] Registration and storage of user information

[0807] The user performs initial setup using the tablet device. First, they enter the SSID and password to connect to their home Wi-Fi. Next, they enter personal information such as their name, age, and address. The entered information is sent from the device to a server, and the server registers the received information in a database. The software used in this process must support a basic user input form and SSL / TLS encrypted communication.

[0808] Email Analysis

[0809] The device monitors the user's email account and sends received emails to a server. The server uses machine learning algorithms to analyze the email content. This analysis process includes detecting fraud-related keywords and verifying sender information. If the analysis results indicate a suspected fraud, the server generates an alert and sends it to the device. The device then displays the alert to the user.

[0810] Examples:

[0811] When a user receives a fraudulent email from the email address "example@example.com," the device sends the contents of the email to the server. The server analyzes the email using a machine learning algorithm that includes the word "fraud." If the server determines that the email is suspected to be fraudulent, it displays a message on the device saying, "This email is suspected to be fraudulent. Do not open it."

[0812] Audio analysis of call content

[0813] When a user receives a suspected fraudulent call on their landline or smartphone, the device (smart speaker or smart earphone) records the call in real time. The audio data is sent to a server, which converts it into text. A machine learning algorithm is then used to determine whether the call is suspected to be fraudulent. If a fraudulent call is suspected, the device sends an alert to the user.

[0814] Examples:

[0815] When a user receives a "scam call pretending to be a relative" on their landline, the smart speaker records the call and sends the data to a server. The server converts the call, including keywords such as "money" and "transfer," into text and analyzes it. If the server determines that the call may be fraudulent, the device notifies the user, "This is a potential fraudulent call. Please hang up immediately."

[0816] Mail analysis

[0817] The user takes a photo of a document, such as mail, with the tablet's camera. The device then sends the captured image data to a server. The server then uses image processing technology to extract text from the image and analyzes the extracted text to determine whether there is any suspicion of fraud. If there is any suspicion of fraud, the device sends an alert to the user.

[0818] Examples:

[0819] A user takes a photo of fraudulent mail delivered to their home using the camera on their tablet device. The device sends the image data to a server, which then uses OCR technology to extract text such as "bank" and "transfer" from the document. If the analysis results indicate a suspicion of fraud, the device notifies the user, saying, "This mail is suspected to be fraudulent. Please be careful."

[0820] Sending alerts

[0821] If fraud is suspected, the server generates an alert message and sends it to the device, which then displays the message on the screen of the smart speaker or tablet and, if necessary, notifies the smart earphones via voice notification.

[0822] Examples:

[0823] Based on the result that "fraud is suspected," the server sends an alert message to the device stating "Warning: Possible fraud." The device displays the message on the tablet screen and uses smart earphones to notify the user by voice, "Possible fraud. Please be careful."

[0824] This system can prevent fraud committed through multiple communication methods against users, including the elderly. By using these methods, users are protected from the threat of fraud and can go about their daily lives with peace of mind.

[0825] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0826] Divide the program processing flow into processing steps

[0827] Registration and storage of user information

[0828] Step 1:

[0829] The user opens the Wi-Fi connection settings screen on the tablet device and enters the SSID and password to establish a connection.

[0830] Input: Wi-Fi SSID and password

[0831] Output: Wi-Fi connection establishment status

[0832] Step 2:

[0833] The user enters personal information (name, age, address, etc.) using a tablet device.

[0834] Input: Personal information such as name, age, address, etc.

[0835] Output: Temporarily save the entered personal information

[0836] Step 3:

[0837] The terminal encrypts the information entered and sends it to the server using the SSL / TLS protocol.

[0838] Input: User personal information

[0839] Output: Encrypted transmission of personal information

[0840] Step 4:

[0841] The server decrypts the received information and stores it in a secure database.

[0842] Input: Encrypted personal information

[0843] Output: Personal information stored in the database

[0844] Email Analysis

[0845] Step 1:

[0846] The device periodically checks the user's email account for new emails.

[0847] Input: Email account information

[0848] Output: New emails

[0849] Step 2:

[0850] The device receives new email and retrieves its contents (body, sender, attachments, etc.).

[0851] Input: New email

[0852] Output: Retrieved email content

[0853] Step 3:

[0854] The device encrypts the email content and sends it to the server.

[0855] Input: Email content

[0856] Output: Encrypted email content

[0857] Step 4:

[0858] The server decrypts the received email and analyzes it using machine learning algorithms.

[0859] Input: Encrypted email content

[0860] Output: Analysis results

[0861] Step 5:

[0862] If fraud is suspected, the server generates an alert message and sends it to the terminal.

[0863] Input: Analysis results

[0864] Output: Alert message

[0865] Step 6:

[0866] The device displays an alert to the user.

[0867] Input: Alert message

[0868] Output: User notification

[0869] Audio analysis of call content

[0870] Step 1:

[0871] Users receive suspicious fraudulent calls on their landline or smartphone.

[0872] Input:Incoming call

[0873] Output: Call Start

[0874] Step 2:

[0875] The device records the call in real time.

[0876] Input: Call content

[0877] Output: Recorded audio data

[0878] Step 3:

[0879] The device sends the recorded audio data to the server.

[0880] Input: Recorded audio data

[0881] Output: Transmitted audio data

[0882] Step 4:

[0883] The server converts the received voice data into text.

[0884] Input: Audio data

[0885] Output: The converted text

[0886] Step 5:

[0887] The server uses machine learning algorithms to analyze the text data.

[0888] Input: Translated text

[0889] Output: Analysis results

[0890] Step 6:

[0891] If fraud is suspected, the server generates an alert message and sends it to the terminal.

[0892] Input: Analysis results

[0893] Output: Alert message

[0894] Step 7:

[0895] The device notifies the user of the alert.

[0896] Input: Alert message

[0897] Output: User notification

[0898] Mail analysis

[0899] Step 1:

[0900] The user takes a photo of the mail item using the camera on the tablet device.

[0901] Input: Mail

[0902] Output: Image data

[0903] Step 2:

[0904] The terminal transmits the acquired image data to the server.

[0905] Input: Image data

[0906] Output: Image data sent

[0907] Step 3:

[0908] The server extracts the text from the image.

[0909] Input: Image data

[0910] Output: Extracted text

[0911] Step 4:

[0912] The server analyzes the extracted text using machine learning algorithms.

[0913] Input: Extracted text

[0914] Output: Analysis results

[0915] Step 5:

[0916] If fraud is suspected, the server generates an alert message and sends it to the terminal.

[0917] Input: Analysis results

[0918] Output: Alert message

[0919] Step 6:

[0920] The device notifies the user of the alert.

[0921] Input: Alert message

[0922] Output: User notification

[0923] Sending alerts

[0924] Step 1:

[0925] If fraud is suspected, the server generates an alert message.

[0926] Input: Analysis results

[0927] Output: Alert message

[0928] Step 2:

[0929] The server sends an alert message to the terminal.

[0930] Input: Alert message

[0931] Output: Alert message sent

[0932] Step 3:

[0933] The device will display an alert message on the screen of a smart speaker or tablet, and if necessary, will also notify the smart earphones via voice.

[0934] Input: The alert message sent

[0935] Output: User notification

[0936] This makes it possible to comprehensively detect fraudulent activities carried out through multiple communication means and quickly warn users.

[0937] (Application example 1)

[0938] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0939] In today's world, where users, including the elderly, are at increased risk of falling victim to special frauds, there is a need to prevent fraudulent acts via email, phone calls, and mail. However, it is difficult for users themselves to detect the signs of these frauds, so an efficient fraud prevention system is needed. Furthermore, conventional systems have had issues with the difficulty of real-time monitoring and highly accurate judgment.

[0940] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0941] In this invention, the server includes means for analyzing received emails and determining whether fraud is suspected, means for acquiring and analyzing phone call content and determining whether fraud is suspected, means for scanning documents with a photographing device and determining whether fraud is suspected, means for sending an alert to a user based on the determination result using a generative AI model, and means for generating a prompt message. This allows users to detect the risk of fraud via email, phone calls, or mail in real time and respond quickly.

[0942] "Means for analyzing received e-mails" refers to a function that analyzes the contents of e-mails received by the system and determines whether there is any suspicion of fraud.

[0943] "Means for acquiring and analyzing call content" refers to a function that acquires the content of a user's call, analyzes that content in real time, and determines whether there is any suspicion of fraud.

[0944] "Means for reading documents with a photographic device" refers to a function that allows a user to photograph a specific document with a photographic device such as a camera, and then determine whether or not there is suspicion of fraud based on the image data.

[0945] A "generative AI model" is an algorithmic model trained using machine learning and deep learning techniques that is used to analyze emails, calls, and documents to identify fraud.

[0946] "Means for generating prompts" refers to a function that uses an AI model to create and present appropriate warning messages or alerts to users based on specific conditions or judgment results.

[0947] "Means for sending alerts" refers to a function that sends an alert or message to a user's device to warn them if fraud is suspected.

[0948] As an embodiment of the present invention, a comprehensive system for protecting users, particularly the elderly, from special frauds is provided. The system includes means for automatically analyzing received emails, phone call contents, mail, etc., determining whether there is any suspected fraud, and sending an alert to the user.

[0949] Registration and storage of user information

[0950] During the initial setup, the user uses a tablet or smartphone to enter information (SSID and password) to connect to their home Wi-Fi. They also enter personal information such as their name, age, and address, and the device sends this information to the server. The server receives the information and registers it in a database.

[0951] Email Analysis

[0952] The email analysis method involves the device monitoring the user's email account and sending received emails to a server. The server then uses a generative AI model to analyze the email content and determine whether it is suspected of being fraudulent. If the analysis results indicate a suspicion of fraud, an alert is sent to the device.

[0953] Audio analysis of call content

[0954] When a user receives a suspected fraudulent call on their landline or smartphone, the device (smart speaker or smart earphone) records the call in real time and sends the audio data to a server. The server converts the audio data into text and uses a generative AI model to determine whether the call is suspected to be fraudulent. If fraud is suspected, the device sends the user an alert saying, "This is a potential fraudulent call. Please hang up immediately."

[0955] Mail analysis

[0956] A user takes a photo of a document, such as mail, with the camera on their tablet or smartphone. The device then sends the captured image data to a server. The server extracts text from the image and analyzes it using a generative AI model to determine whether it is suspected of fraud. If fraud is suspected, the device sends the user an alert saying, "This mail is suspected to be fraudulent. Please be careful."

[0957] Sending alerts

[0958] If the server determines that fraud is suspected, it generates a prompt message and sends it to the device, which then displays the message on the screen of the smart speaker or smartphone and, if necessary, sends a voice notification to the smart earphones.

[0959] Hardware and software used

[0960] Hardware: smartphones, tablets, smart speakers, smart earphones, cameras, microphones

[0961] Software: OCR software (e.g., tesseract-ocr), machine learning models (e.g., scikit-learn), speech recognition software (e.g., Google Speech Recognition API), email processing libraries (e.g., python-email, smtplib), call monitoring software (e.g., Twilio API)

[0962] Prompt Sentence Examples

[0963] Use the following as an example prompt for analyzing suspected fraudulent emails:

[0964] "We have come across a potentially fraudulent email. The contents are as follows:

[0965] From: {sender}

[0966] Subject: {subject}

[0967] Contents: {body}

[0968] Please be careful as this appears to be a scam."

[0969] This will enable users to respond to fraudulent activity in real time, improving safety.

[0970] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0971] Step 1: Register and save user information

[0972] During the initial setup, a user uses a tablet or smartphone to enter information (SSID and password) to connect to their home Wi-Fi. They also enter personal information such as their name, age, and address, which the device then sends to the server. The server receives the information and registers it in a database. The input here is the user's Wi-Fi information and personal information, and the output is the user information stored in the database.

[0973] Step 2: Capture and parse emails

[0974] The device monitors the user's email account and sends received emails to a server. The server uses a generative AI model to analyze the email content and determine whether it is suspected of fraud. The input here is the content of the received email, and the output is a judgment result on whether it is suspected of fraud.

[0975] Step 3: Audio analysis of the call

[0976] When a user receives a suspected fraudulent call on their landline or smartphone, the device (smart speaker or smart earphone) records the call in real time and sends the audio data to a server. The server converts the audio data into text and uses a generative AI model to determine whether the call is suspected of fraud. The input here is the recorded call content, and the output is the converted text and a determination of whether the call is suspected of fraud.

[0977] Step 4: Image analysis of mail

[0978] A user takes a photo of a document, such as mail, using a tablet or smartphone camera. The device then sends the captured image data to a server. The server extracts text from the image and uses a generative AI model to analyze the text and determine whether it is suspected of fraud. The input here is the image of the mail, and the output is the extracted text and a determination of whether it is suspected of fraud.

[0979] Step 5: Sending an alert

[0980] If the server determines that fraud is suspected, it generates a prompt and an alert message and sends it to the device. The device then displays the warning message on the screen of the smart speaker or smartphone, and if necessary, notifies the smart earphones by voice. The input here is the judgment result regarding the suspected fraud, and the output is the generated prompt and alert message.

[0981] Through these steps, users can detect fraud risks in real time and respond quickly.

[0982] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0983] As an embodiment for carrying out the present invention, various means of a system (hereinafter referred to as an improved system) that combines a special fraud prevention system with an emotion engine will be described.

[0984] Registration and storage of user information

[0985] During the initial setup, the user uses the tablet device to enter the SSID and password to connect to their home Wi-Fi. After that, they enter personal information such as their name, age, and address, and the device sends this information to the server. The server receives the information and registers it in a database.

[0986] Email analysis and emotion recognition

[0987] The device monitors the user's email account and sends received emails to a server. The server analyzes the email content and determines whether it is suspected of being fraudulent. It also recognizes the user's emotions based on the email content. If the analysis results indicate a suspicion of fraud, it sends an alert to the device.

[0988] Voice analysis and emotion recognition of call content

[0989] When a user receives a suspected fraudulent call on their landline or smartphone, the smart speaker or earphone device records the call in real time and sends the audio data to a server. The server converts the audio data into text and then analyzes the text to determine whether the call is suspected of fraud. The system also recognizes the user's emotions from the call content and sends an alert if there is a change in emotion.

[0990] Mail analysis and emotion recognition

[0991] The user takes a photo of a document, such as mail, using the tablet's camera. The device then sends the captured image data to a server. The server extracts text from the image and analyzes it to determine whether it is suspected of fraud. In addition, the system recognizes the user's emotions based on the document's content. If it determines that the document is suspected of fraud, the device sends an alert to the user.

[0992] Sending alerts

[0993] If fraud is suspected, the server generates an alert message and sends it to the device. The device then displays the message on the screen of the smart speaker or tablet, and if necessary, sends a voice notification to the smart earphones. The emotion engine also notifies the user of the analysis results and provides appropriate advice based on the user's psychological state.

[0994] Specific examples

[0995] Specific examples of mail analysis

[0996] 1. The user takes a photo of mail delivered to their home with their camera.

[0997] 2. The device sends the image data to the server.

[0998] 3. The server extracts text from the image and analyzes the text using machine learning algorithms.

[0999] 4. The server further recognizes the user's emotions based on the document content.

[1000] 5. If it determines that there is a suspicion of fraud, the device will send the user an alert saying, "This mail is suspected to be fraudulent. Please be careful," and will also send an emotional alert saying, "This mail conveys a sense of anxiety. Please ignore it."

[1001] Specific examples of call analysis

[1002] 1. A user receives a suspected fraudulent call on their landline.

[1003] 2. The smart speaker device records the call and sends the data to the server.

[1004] 3. The server converts the speech into text and analyzes it.

[1005] 4. The server recognizes the user's emotions from the content of the call and sends an alert if there is a change in emotions.

[1006] 5. If fraud is suspected, the device will send the user an alert saying, "This is a potential fraudulent call. Please hang up immediately," and will also provide an emotional alert saying, "You are feeling surprised and anxious. Please hang up."

[1007] As a result, the improved system can prevent the threat of special fraud and ensure safety for users, including the elderly. In addition, it can recognize the user's emotional state in real time and provide psychological support.

[1008] The processing flow will be explained below.

[1009] Email analysis and emotion recognition

[1010] Step 1:

[1011] The user receives an email.

[1012] Step 2:

[1013] The terminal acquires the contents of the received email and sends them to the server.

[1014] Step 3:

[1015] The server analyzes the content of the email to determine whether it is suspected of being fraudulent.

[1016] Step 4:

[1017] The server recognizes the user's emotions based on the email content.

[1018] Step 5:

[1019] The server sends the analysis results and emotion recognition results to the device.

[1020] Step 6:

[1021] If fraud is suspected, the device will display or notify the user of an alert and an emotion alert.

[1022] Voice analysis and emotion recognition of call content

[1023] Step 1:

[1024] A user initiates a call on a desk phone or smartphone.

[1025] Step 2:

[1026] The device, a smart speaker or smart earphones, records the call in real time.

[1027] Step 3:

[1028] The device sends the recorded audio data to the server.

[1029] Step 4:

[1030] The server converts the audio data into text.

[1031] Step 5:

[1032] The server analyzes the converted text to determine whether it is suspected of fraud.

[1033] Step 6:

[1034] The server recognizes the user's emotions based on the contents of the call.

[1035] Step 7:

[1036] The server sends the analysis results and emotion recognition results to the device.

[1037] Step 8:

[1038] If fraud is suspected, the device will display an alert and emotion alert to the user or notify them audibly during the call.

[1039] Mail analysis and emotion recognition

[1040] Step 1:

[1041] The user takes a photo of the mail that arrives at their home using the camera on their tablet device.

[1042] Step 2:

[1043] The terminal transmits the captured image data to the server.

[1044] Step 3:

[1045] The server extracts the text from the image.

[1046] Step 4:

[1047] The server analyzes the extracted text to determine whether it is suspected of fraud.

[1048] Step 5:

[1049] The server recognizes the user's emotions based on the document content.

[1050] Step 6:

[1051] The server sends the analysis results and emotion recognition results to the device.

[1052] Step 7:

[1053] If fraud is suspected, the device will display or notify the user of an alert and an emotion alert.

[1054] account settings

[1055] Step 1:

[1056] The user opens the initial setup screen using the tablet device.

[1057] Step 2:

[1058] The user enters the Wi-Fi SSID and password.

[1059] Step 3:

[1060] The device attempts to connect to Wi-Fi and displays the results to the user.

[1061] Step 4:

[1062] The user enters personal information such as name, age, and address.

[1063] Step 5:

[1064] The terminal transmits the input information to the server.

[1065] Step 6:

[1066] The server receives the information and stores it in a database.

[1067] Step 7:

[1068] The server sends a confirmation message to the terminal that the save has been completed, and the terminal notifies the user.

[1069] Through these processing steps, the improved system can ensure the safety of users and prevent them from committing fraud. Furthermore, by utilizing the emotion engine, it is possible to determine the user's emotional state in real time and provide psychological support.

[1070] Example 2

[1071] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1072] In modern society, special frauds are becoming increasingly sophisticated and their methods are diversifying. Fraud targeting the elderly in particular has become a major social problem, and conventional fraud prevention systems are often unable to keep up with new methods. Furthermore, the lack of appropriate countermeasures and support that take into account the user's emotional state makes it difficult to prevent damage before it occurs. To solve this problem, it is necessary to provide a system that can effectively analyze communication methods (email, phone calls, documents) that are suspected of being fraudulent, and that can recognize and respond to the user's emotional state in real time.

[1073] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1074] In this invention, the server includes means for analyzing received electronic messages and determining whether fraud is suspected, means for acquiring and analyzing the contents of phone calls and determining whether fraud is suspected, means for photographing and reading documents and determining whether fraud is suspected, means for analyzing the emotional state of the user, and means for sending an alert to the user based on the determination result and the emotional state. This makes it possible to deal with a variety of fraudulent methods, provide appropriate support that takes into account the psychological state of the user, and prevent fraud damage before it occurs.

[1075] "Means for analyzing received electronic messages to determine whether they are suspected of fraud" refers to a combination of software and hardware that processes the content of emails and messages received by a user and automatically detects possible fraudulent activity.

[1076] "Means for acquiring and analyzing call content to determine whether fraud is suspected" refers to a combination of software and hardware that records the user's call content and analyzes the audio data to identify possible fraud.

[1077] "Means for photographing and reading documents to determine whether they are suspected of fraud" refers to a combination of software and hardware that converts the contents of a document photographed by a user with a camera into text data using technologies such as optical character recognition, and then analyzes the text data to determine whether it is suspected of fraud.

[1078] The "means for analyzing a user's emotional state" is a combination of software and hardware for analyzing a user's emotional state based on the contents of electronic messages, conversations and documents, and evaluating the results.

[1079] The "means for sending an alert to the user based on the judgment result and emotional state" refers to a combination of software and hardware that, when a suspected fraud is judged to be present, generates a warning message that takes into account the judgment result and the user's emotional state and notifies the user.

[1080] "Means for registering and storing user information" refers to a combination of software and hardware for registering users' personal information in a database and using or referencing it when necessary.

[1081] "Means for monitoring a user's voice communications in real time and sending an alert if the communications contain suspected fraudulent content" refers to a combination of software and hardware that monitors a user's calls in real time and immediately issues an alert if suspected fraudulent content is detected.

[1082] As an embodiment of the present invention, the detailed configuration and operation of a system (hereinafter referred to as "this system") that combines a special fraud prevention system with an emotion recognition engine will be described.

[1083] Registration and storage of user information

[1084] The user uses the tablet device for initial setup and enters the SSID and password to connect to their home Wi-Fi. Next, they enter personal information such as their name, age, and address into the device. This information is sent from the device to the server. The server registers the received information in a database, which allows the user's basic information to be safely stored.

[1085] Analysis of electronic messages and emotion recognition

[1086] The device periodically monitors the user's email account and receives new emails. The received emails are immediately sent to the server. The server uses a text analysis engine to analyze the email content and determine whether it is suspected of being fraudulent. The server also uses an emotion recognition engine to extract the user's emotions from the email content. If it determines that the email is suspected of being fraudulent, it sends an alert message to the device and displays a warning to the user.

[1087] Voice analysis and emotion recognition of call content

[1088] When a user receives a suspected fraudulent call on their landline or smartphone, the smart speaker or earphones that are part of the device record the call in real time. The recorded data is then immediately sent to a server. The server uses a speech recognition engine to convert the voice data into text and analyzes the text to determine whether the call is suspected of fraud. The server also recognizes the user's emotions from the call content and sends an alert if a change in emotion is detected.

[1089] Document Analysis and Emotion Recognition

[1090] The user takes a photo of a document, such as mail, with the tablet's camera. The device then sends the captured image data to the server. The server uses an image analysis engine to extract text from the image and analyzes that text to determine whether it is suspected of fraud. The server then recognizes the user's emotions based on the document's content. If it determines that the document is suspected of fraud, the device sends an alert to the user.

[1091] Sending alerts

[1092] If a suspected fraud attempt is detected, the server generates an alert message and sends it to the device, which then displays the message on the screen of the smart speaker or tablet and, if necessary, also notifies the smart earphones via voice notification, allowing the user to immediately become aware of the possibility of fraud.

[1093] Specific examples

[1094] Specific examples of mail analysis

[1095] 1. The user takes a photo of the mail that arrives at their home with their camera.

[1096] 2. The device sends the image data to the server.

[1097] 3. The server extracts text from the image and analyzes the text using machine learning algorithms.

[1098] 4. The server further recognizes the user's emotions based on the document content.

[1099] 5. If it determines that there is a suspicion of fraud, the device will send the user an alert saying, "This mail is suspected to be fraudulent. Please be careful," and will also send an emotional alert saying, "This mail conveys a sense of anxiety. Please ignore it."

[1100] Specific examples of call analysis

[1101] 1. A user receives a suspected fraudulent call on their landline.

[1102] 2. The smart speaker device records the call and sends the data to the server.

[1103] 3. The server converts the speech into text and analyzes it.

[1104] 4. The server recognizes the user's emotions from the content of the call and sends an alert if there is a change in emotion.

[1105] 5. If fraud is suspected, the device will send the user an alert saying, "This is a potential fraudulent call. Please hang up immediately," and will also provide an emotional alert saying, "You are feeling surprised and anxious. Please hang up."

[1106] In this way, this system can prevent the threat of special fraud and ensure the safety of users, including the elderly. In addition, it can recognize the user's emotional state in real time and provide psychological support.

[1107] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1108] Registration and storage of user information

[1109] Processing Steps

[1110] Step 1:

[1111] The user opens the settings screen on the tablet device and enters the SSID and password to connect to their home Wi-Fi.

[1112] Input: SSID, password

[1113] Output: Wi-Fi connection complete message

[1114] Specific operation: The user enters the SSID and password in the specified fields and presses the "Connect" button. The device receives this and attempts to connect to Wi-Fi. If the connection is successful, a connection completion message is displayed.

[1115] Step 2:

[1116] The user enters personal information such as name, age, and address into the terminal.

[1117] Input: Personal information such as name, age, address, etc.

[1118] Output: Input confirmation screen

[1119] Specific operation: The user enters personal information into each field and presses the "Submit" button. The device retrieves this information and displays a confirmation screen.

[1120] Step 3:

[1121] The terminal transmits the entered personal information to the server.

[1122] Input: Personal information entered on the device

[1123] Output: Message that data has been sent to the server

[1124] Specific operation: The terminal packetizes the entered personal information and sends it to the server using a secure communication protocol (e.g., HTTPS).

[1125] Step 4:

[1126] The server registers the received personal information in a database.

[1127] Input: Personal information sent from the device

[1128] Output: Database registration completion message

[1129] Specific operation: The server analyzes the received data and stores it in the database appropriately. Once the data has been stored, it returns a registration completion message to the device.

[1130] Analysis of electronic messages and emotion recognition

[1131] Processing Steps

[1132] Step 1:

[1133] The device periodically monitors the user's email account and receives new emails.

[1134] Input: Email login details

[1135] Output: New email information

[1136] Specific operation: The device sends a query to the mail server at the specified interval to check for new mail. If new mail is found, the device retrieves its contents.

[1137] Step 2:

[1138] The device sends the contents of the new email to the server.

[1139] Input: New email content

[1140] Output: Message that email has been sent to the server

[1141] Specific operation: The device obtains the body and header information of the new email and sends it to the server.

[1142] Step 3:

[1143] The server uses a text analysis engine to analyze the email content and determine whether it is suspected of being fraudulent.

[1144] Input: Email text data

[1145] Output: Fraud detection result

[1146] What it does: The server uses a text analysis engine (e.g., an NLP model) to analyze the content of the email and determine whether it is fraudulent. The result is stored as a flag.

[1147] Step 4:

[1148] The server uses an emotion recognition engine to extract the user's emotion from the content of the email.

[1149] Input: Email text data

[1150] Output: Emotion recognition result

[1151] Specific operation: The server uses an emotion recognition engine (e.g., emotion analysis API) to extract the user's emotion from the email body and saves the result as a flag.

[1152] Step 5:

[1153] If fraud is suspected, an alert message will be sent to the terminal and a warning will be displayed to the user.

[1154] Input: Fraud detection result, emotion recognition result

[1155] Output: Alert message screen

[1156] Specific operation: If the fraud flag is set, the server generates an alert message and sends it to the terminal. The terminal receives it and displays the warning message to the user.

[1157] Voice analysis and emotion recognition of call content

[1158] Processing Steps

[1159] Step 1:

[1160] Users receive suspicious fraudulent calls on their landline or smartphone.

[1161] Input: Incoming call notification

[1162] Output: Call start signal

[1163] Specific operation: The user receives an incoming call and answers the phone. The device detects this and transitions to a call state.

[1164] Step 2:

[1165] Smart speakers and smart earphones that are part of the device record the contents of calls in real time.

[1166] Input: Call audio data

[1167] Output: Recording data

[1168] Specific operation: The terminal starts recording as soon as the call starts and stores the audio data in a buffer.

[1169] Step 3:

[1170] The recording data is immediately sent to the server.

[1171] Input: Recording data

[1172] Output: Message that data has been sent to the server

[1173] What happens: The device sends the recording data to the server using a secure protocol (e.g., HTTPS).

[1174] Step 4:

[1175] The server uses a speech recognition engine to convert the voice data into text.

[1176] Input: Audio data

[1177] Output: Text data

[1178] Specific operation: The server uses a speech recognition engine (e.g., Speech-to-Text API) to convert the voice data into text.

[1179] Step 5:

[1180] The server analyzes the text to determine if it is suspected of fraud.

[1181] Input: Converted text data

[1182] Output: Fraud detection result

[1183] What it does: The server uses a text analysis engine to determine the likelihood of fraud, and stores the result as a flag.

[1184] Step 6:

[1185] The server recognizes the user's emotions from the content of the call and sends an alert if there is a change in emotion.

[1186] Input: Text data

[1187] Output: Emotion recognition result

[1188] Specific operation: The server uses an emotion recognition engine to extract the user's emotion from the text data and saves the result as a flag.

[1189] Step 7:

[1190] If fraud is suspected, an alert message will be sent to the terminal and a warning will be displayed to the user.

[1191] Input: Fraud detection result, emotion recognition result

[1192] Output: Alert message screen

[1193] Specific operation: If the fraud flag is set, the server generates an alert message and sends it to the terminal. The terminal receives it and displays the warning message to the user.

[1194] Document Analysis and Emotion Recognition

[1195] Processing Steps

[1196] Step 1:

[1197] The user takes a photo of the mail item using the camera on the tablet device.

[1198] Input: Image data of mail

[1199] Output: Shooting completion message

[1200] Specific operation: The user takes a photo of the mail using the camera function of the tablet device. The image data is stored on the device.

[1201] Step 2:

[1202] The terminal transmits the image data to the server.

[1203] Input: Image data of mail

[1204] Output: Message that data has been sent to the server

[1205] Specific operation: The device sends the acquired image data to the server using a secure protocol (e.g., HTTPS).

[1206] Step 3:

[1207] The server uses an image analysis engine to extract text from the image.

[1208] Input: Image data

[1209] Output: Extracted text data

[1210] Specific operation: The server uses an image analysis engine (e.g., OCR API) to extract text data from the image.

[1211] Step 4:

[1212] The server analyzes the text to determine if it is suspected of fraud.

[1213] Input: Extracted text data

[1214] Output: Fraud detection result

[1215] What it does: The server uses a text analysis engine to determine the likelihood of fraud, and stores the result as a flag.

[1216] Step 5:

[1217] The server recognizes the user's emotions based on the document content.

[1218] Input: Extracted text data

[1219] Output: Emotion recognition result

[1220] Specific operation: The server uses an emotion recognition engine to extract the user's emotion from the document content, and the result is saved as a flag.

[1221] Step 6:

[1222] If fraud is suspected, an alert message will be sent to the terminal and a warning will be displayed to the user.

[1223] Input: Fraud detection result, emotion recognition result

[1224] Output: Alert message screen

[1225] Specific operation: If the fraud flag is set, the server generates an alert message and sends it to the terminal. The terminal receives it and displays a warning message to the user.

[1226] Sending alerts

[1227] Processing Steps

[1228] Step 1:

[1229] The server detects suspected fraud.

[1230] Input: Fraud detection result, emotion recognition result

[1231] Output: Alert message

[1232] Specific operation: The server checks the fraud determination flag and emotion recognition flag and generates an alert message if necessary.

[1233] Step 2:

[1234] The server generates an alert message and sends it to the terminal.

[1235] Input: Fraud detection result, emotion recognition result

[1236] Output: The generated alert message

[1237] Specific operation: The server generates an alert message and sends it to the terminal in an appropriate format.

[1238] Step 3:

[1239] The device displays or notifies you of an alert message.

[1240] Input: The generated alert message

[1241] Output: The displayed warning message screen

[1242] Specific operation: When the terminal receives an alert message, it displays a visual or audio warning message to the user.

[1243] Through the above processing steps, the system can provide high detection accuracy and rapid response to a variety of fraudulent methods, while also providing appropriate support based on the user's emotional state.

[1244] (Application example 2)

[1245] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1246] In recent years, there has been an increase in special frauds targeting the elderly, and many of these are committed via email, telephone, or mail. This has led to a growing need for systems that can not only detect fraud but also monitor users' emotions in real time and provide appropriate responses based on their psychological state. While conventional systems specialize in detecting fraud, they do not provide support that takes users' emotions into consideration, resulting in a lack of psychological support.

[1247] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1248] In this invention, the server includes means for analyzing received electronic documents and determining whether fraud is suspected, means for acquiring and analyzing the contents of phone calls and determining whether fraud is suspected, means for scanning documents with an imaging device and determining whether fraud is suspected, means for recognizing the user's emotions using an emotion recognition engine when determining whether fraud is suspected, and providing appropriate advice on how to respond if there is a possibility of fraud, and means for sending a warning to the user based on the determination result. This makes it possible not only to respond immediately to information that is suspected of being fraud, but also to provide psychological support that takes into account the user's emotional state.

[1249] An "electronic document" is a document stored in digital format, including email, PDF, Word files, etc.

[1250] "Call content" refers to information exchanged through voice communication, such as the content of a conversation conducted via telephone or internet telephone.

[1251] "Photographing device" refers to equipment for taking still or moving images, and includes devices with camera functions.

[1252] An "emotion recognition engine" is a combination of software and hardware for identifying and analyzing human emotional states.

[1253] A "warning" is a message or notification that warns the user of a caution or danger based on a specific condition.

[1254] "User Information" refers to personal data relating to users of the system, including name, age, address, contact details, etc.

[1255] "Analysis" refers to the process of examining a set of data or information in detail to understand its content, trends, and relationships.

[1256] "Judgment" is the act of determining an outcome based on collected data and information according to specific conditions and criteria.

[1257] System Overview

[1258] This system is a security system that combines fraud prevention and user emotion recognition. The system analyzes the contents of received electronic documents, phone calls, and documents captured using a camera, and issues a warning if fraud is suspected. Furthermore, the system uses an emotion recognition engine to analyze the user's emotional state and provide appropriate responses.

[1259] Hardware and software used

[1260] Hardware:

[1261] Smartphone

[1262] camera

[1263] software:

[1264] Server side: Django

[1265] Emotion Recognition API:Azure Cognitive Services Emotion API

[1266] NLP libraries: spaCy, nltk

[1267] Deep Learning: TensorFlow, PyTorch

[1268] Speech Recognition: Google Speech-to-Text API

[1269] OCR library: Tesseract

[1270] Program processing

[1271] 1. Electronic Document Analysis:

[1272] The server periodically retrieves emails received via the API and performs text analysis using an NLP library (spaCy or nltk).

[1273] It analyzes text structure and keywords, and if there is a possibility of fraud, it uses the Azure Cognitive Services Emotion API to analyze the user's emotions.

[1274] If fraud is suspected, a warning message will be generated and sent to the user's smartphone.

[1275] 2. Call analysis:

[1276] It monitors smartphone calls in real time and converts audio data into text using the Google Speech-to-Text API.

[1277] Use NLP libraries to analyze text and determine potential fraud.

[1278] The Azure Cognitive Services Emotion API analyzes the user's emotions, and if anxiety or surprise is detected, a warning message and emotion-based advice is sent to the user.

[1279] 3. Document Analysis:

[1280] A photograph of a document such as mail is taken using a photographing device (smartphone camera), and the image data is sent to the server.

[1281] The server uses an OCR library (Tesseract) to extract text from image data and performs text analysis using an NLP library.

[1282] If fraud is suspected, emotion recognition is performed, a message advising appropriate action is generated, and a warning is sent to the user's smartphone.

[1283] Specific examples

[1284] Specific examples of email analysis

[1285] Process flow: Retrieve email from SMTP server → Analyze text using spaCy → Analyze emotions using Azure Cognitive Services Emotion API → Warn user if fraud is suspected

[1286] Example alerts: "This email may be fraudulent. Please be cautious.", "This email makes me feel uneasy. Please ignore it."

[1287] Prompt Sentence Examples

[1288] "Please analyze the content of the email below to determine the likelihood of a scam and the associated emotions.

[1289] The email reads: "Dear customer, you have been selected for our special campaign. Please update your account information to receive your prize."

[1290] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1291] Step 1: Obtaining electronic documents

[1292] Description: The server periodically retrieves the user's email via API.

[1293] Input: User's email account information

[1294] Output: Retrieved email data

[1295] What happens: The server fetches emails through the SMTP server's API and retrieves new emails in the inbox.

[1296] Step 2: Text analysis

[1297] Description: The server performs text analysis on the body of the retrieved email.

[1298] Input: Email data

[1299] Output: Analysis results (suspected fraud)

[1300] What it does: The server uses an NLP library (e.g. spaCy) to tokenize the email body, extract keywords, and perform contextual analysis to assess the likelihood of fraud.

[1301] Step 3: Sentiment Analysis

[1302] Description: The server analyzes the user's emotional state based on the content of the email.

[1303] Input: Email data, NLP analysis results

[1304] Output: Emotion analysis results

[1305] How it works: The server uses the Azure Cognitive Services Emotion API to estimate the emotions the user may be feeling from the content of the email.

[1306] Step 4: Generate warning messages

[1307] Description: If fraud is suspected, the server generates a warning message.

[1308] Input: NLP analysis results, sentiment analysis results

[1309] Output: Warning message

[1310] Specific behavior: The server generates a warning message based on the analysis results, for example, "This email may be fraudulent. Please be careful."

[1311] Step 5: Sending an alert

[1312] Description: The server sends the generated warning message to the user's terminal.

[1313] Input: warning message

[1314] Output: A warning displayed on the user's terminal

[1315] Specific operation: The server sends a warning message to the user's smartphone via push notification.

[1316] Step 6: Capture the call

[1317] Description: The device receives the user's call content in real time.

[1318] Input: Call audio data

[1319] Output: Recorded audio data

[1320] How it works: The smartphone uses its built-in microphone to record the conversation and sends it to the server as audio data.

[1321] Step 7: Speech to Text

[1322] Description: The server converts the call audio data into text.

[1323] Input: Captured audio data

[1324] Output: Text data

[1325] Specific operation: The server uses the Google Speech-to-Text API to convert the captured voice data into text.

[1326] Step 8: Call Analysis

[1327] Description: The server analyzes the text of the call to determine if it is suspected of fraud.

[1328] Input: Text data

[1329] Output: Analysis results (suspected fraud)

[1330] What it does: The server uses an NLP library to tokenize the call, extract keywords and perform contextual analysis to assess the likelihood of fraud.

[1331] Step 9: Sentiment Analysis

[1332] Description: The server analyzes the user's emotional state from the content of the call.

[1333] Input: Text data, NLP analysis results

[1334] Output: Emotion analysis results

[1335] Specific operation: The server uses the Azure Cognitive Services Emotion API to estimate the user's emotions from the content of the call.

[1336] Step 10: Generate warning messages

[1337] Description: If fraud is suspected or there is a change in sentiment, the server generates a warning message.

[1338] Input: NLP analysis results, sentiment analysis results

[1339] Output: Warning message

[1340] Specific behavior: Based on the analysis results, the server generates a warning message such as "This call may be fraudulent. Please hang up immediately."

[1341] Step 11: Sending an alert

[1342] Description: The server sends the generated warning message to the user's terminal.

[1343] Input: warning message

[1344] Output: A warning displayed on the user's terminal

[1345] Specific operation: The server sends a warning message to the user's smartphone via push notification.

[1346] Step 12: Photograph the document

[1347] Description: A user takes a picture of a document, such as mail, with a camera.

[1348] Input: Image data

[1349] Output: Photographed document image

[1350] Specific action: The user uses the smartphone camera to take a picture of the mail.

[1351] Step 13: Extracting text from images

[1352] Description: The server extracts text from the captured image.

[1353] Input: Image data

[1354] Output: Extracted text data

[1355] Specific operation: The server uses an OCR library (Tesseract) to extract text from the image.

[1356] Step 14: Document Content Analysis

[1357] Description: The server analyzes the text data to determine if it is suspected of fraud.

[1358] Input: Text data

[1359] Output: Analysis results (suspected fraud)

[1360] How it works: The server uses an NLP library to tokenize the document content, extract keywords and perform contextual analysis to assess the likelihood of fraud.

[1361] Step 15: Sentiment Analysis

[1362] Description: The server analyzes the user's emotional state from the document content.

[1363] Input: Text data, NLP analysis results

[1364] Output: Emotion analysis results

[1365] Specific operation: The server uses the Azure Cognitive Services Emotion API to estimate the user's emotions from the document content.

[1366] Step 16: Generate warning messages

[1367] Description: If fraud is suspected, the server generates a warning message.

[1368] Input: NLP analysis results, sentiment analysis results

[1369] Output: Warning message

[1370] Specific operation: Based on the analysis results, the server creates a message such as "This mail is suspected to be fraudulent. Please be careful."

[1371] Step 17: Sending Alerts

[1372] Description: The server sends the generated warning message to the user's terminal.

[1373] Input: warning message

[1374] Output: A warning displayed on the user's terminal

[1375] Specific operation: The server sends a warning message to the user's smartphone via push notification.

[1376] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1377] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1378] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1379] [Third embodiment]

[1380] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1381] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1382] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1383] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1384] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1385] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1386] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1387] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1388] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1389] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1390] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1391] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1392] As an embodiment of the present invention, various means constituting a special fraud prevention system (hereinafter referred to as the present system) will be described.

[1393] Registration and storage of user information

[1394] During the initial setup, the user uses the tablet device to enter the SSID and password to connect to their home Wi-Fi. After that, they enter personal information such as their name, age, and address, and the device sends this information to the server. The server receives the information and registers it in a database.

[1395] Email Analysis

[1396] The device monitors the user's email account and sends received emails to a server, which analyzes the email content and uses machine learning algorithms to determine whether the email is suspected of being fraudulent. If the analysis results indicate a suspected fraud, an alert is sent to the device.

[1397] Audio analysis of call content

[1398] When a user receives a suspected fraudulent call on their landline or smartphone, the device (smart speaker or smart earphone) records the call in real time and sends the audio data to a server. The server converts the audio data into text and determines whether it is suspected to be fraudulent. If it is, the device sends an alert to the user.

[1399] Mail analysis

[1400] The user takes a photo of a document, such as mail, with the tablet's camera. The device then sends the captured image data to a server. The server extracts text from the image and analyzes it to determine whether it is suspected of fraud. If it is suspected of fraud, the device sends an alert to the user.

[1401] Sending alerts

[1402] If fraud is suspected, the server generates an alert message and sends it to the device, which then displays the message on the screen of the smart speaker or tablet and, if necessary, notifies the smart earphones via voice notification.

[1403] Specific examples

[1404] Specific examples of mail analysis

[1405] 1. The user takes a photo of mail delivered to their home with their camera.

[1406] 2. The device sends the image data to the server.

[1407] 3. The server extracts text from the image and analyzes the text using machine learning algorithms.

[1408] 4. If it is determined that there is a suspicion of fraud, the device will send an alert to the user saying, "This mail is suspected to be fraudulent. Please be careful."

[1409] Specific examples of call analysis

[1410] 1. A user receives a suspected fraudulent call on their landline.

[1411] 2. The smart speaker device records the call and sends the data to the server.

[1412] 3. The server converts the speech into text and analyzes it.

[1413] 4. If fraud is suspected, the device will send an alert to the user saying, "This is a potential fraudulent call. Please hang up immediately."

[1414] In this way, the system can prevent the threat of special frauds and ensure safety for users, including the elderly.

[1415] The processing flow will be explained below.

[1416] Email Analysis

[1417] Step 1:

[1418] The user receives an email.

[1419] Step 2:

[1420] The terminal acquires the contents of the received email and sends them to the server.

[1421] Step 3:

[1422] The server analyzes the content of the email to determine whether it is suspected of being fraudulent.

[1423] Step 4:

[1424] The server sends the analysis results to the device.

[1425] Step 5:

[1426] If fraud is suspected, the device will display an alert or notify the user.

[1427] Audio analysis of call content

[1428] Step 1:

[1429] A user initiates a call on a desk phone or smartphone.

[1430] Step 2:

[1431] The device, a smart speaker or smart earphones, records the call in real time.

[1432] Step 3:

[1433] The device sends the recorded audio data to the server.

[1434] Step 4:

[1435] The server converts the audio data into text.

[1436] Step 5:

[1437] The server analyzes the converted text to determine whether it is suspected of fraud.

[1438] Step 6:

[1439] If fraud is suspected, the device will display an alert to the user or notify them audibly during the call.

[1440] Mail analysis

[1441] Step 1:

[1442] The user takes a photo of the mail that arrives at their home using the camera on their tablet device.

[1443] Step 2:

[1444] The terminal transmits the captured image data to the server.

[1445] Step 3:

[1446] The server extracts the text from the image.

[1447] Step 4:

[1448] The server analyzes the extracted text to determine whether it is suspected of fraud.

[1449] Step 5:

[1450] If fraud is suspected, the device will display an alert or notify the user.

[1451] account settings

[1452] Step 1:

[1453] The user opens the initial setup screen using the tablet device.

[1454] Step 2:

[1455] The user enters the Wi-Fi SSID and password.

[1456] Step 3:

[1457] The device attempts to connect to Wi-Fi and displays the results to the user.

[1458] Step 4:

[1459] The user enters personal information such as name, age, and address.

[1460] Step 5:

[1461] The terminal transmits the input information to the server.

[1462] Step 6:

[1463] The server receives the information and stores it in a database.

[1464] Step 7:

[1465] The server sends a confirmation message to the terminal that the save has been completed, and the terminal notifies the user.

[1466] Through these processing steps, the system is designed to ensure the safety of users and prevent special frauds from occurring.

[1467] Example 1

[1468] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1469] With the development of modern communication methods, frauds using email, telephone, and mail are on the rise. These frauds have become a problem that causes significant economic losses and psychological burdens to many people, including the elderly. Existing countermeasures have limitations, and more effective and comprehensive measures are needed.

[1470] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1471] In this invention, the server includes means for analyzing received emails and determining whether they are suspected of fraud using a machine learning algorithm, means for recording phone calls in real time, converting the audio data into text, and determining whether they are suspected of fraud, and means for scanning documents with a camera, extracting text from the image data, and determining whether they are suspected of fraud. This makes it possible to comprehensively detect fraudulent activities committed through multiple communication means and quickly warn users.

[1472] A "machine learning algorithm" is a computational method that allows a computer to learn patterns from data and make predictions or classifications.

[1473] "Voice data" refers to data that represents call content and voice information in digital format.

[1474] "Convert to text" is the process of converting non-text data, such as audio or images, into written information.

[1475] "Document" refers to information recorded in writing, including, for example, mail, letters, notices, etc.

[1476] A "camera" is a device that takes an image of an object and stores it as digital data.

[1477] "Image data" refers to data in which images taken with a camera or the like are stored in digital format.

[1478] "Text extraction" is the process of extracting textual information from image or audio data.

[1479] A "server" is a computer system that stores and processes data and provides services to clients over a network.

[1480] An "alert" is a notification that alerts or warns the user, and is provided as a voice or text message.

[1481] "User" refers to anyone who uses this system, and is primarily intended to prevent fraud.

[1482] As an embodiment of the present invention, various means of the special fraud prevention system (hereinafter referred to as the "system") will be specifically described. The system has the following main functions: registration and storage of user information, analysis of e-mails, analysis of voice of phone calls, analysis of mail, and sending of alerts.

[1483] Registration and storage of user information

[1484] The user performs initial setup using the tablet device. First, they enter the SSID and password to connect to their home Wi-Fi. Next, they enter personal information such as their name, age, and address. The entered information is sent from the device to a server, and the server registers the received information in a database. The software used in this process must support a basic user input form and SSL / TLS encrypted communication.

[1485] Email Analysis

[1486] The device monitors the user's email account and sends received emails to a server. The server uses machine learning algorithms to analyze the email content. This analysis process includes detecting fraud-related keywords and verifying sender information. If the analysis results indicate a suspected fraud, the server generates an alert and sends it to the device. The device then displays the alert to the user.

[1487] Examples:

[1488] When a user receives a fraudulent email from the email address "example@example.com," the device sends the contents of the email to the server. The server analyzes the email using a machine learning algorithm that includes the word "fraud." If the server determines that the email is suspected to be fraudulent, it displays a message on the device saying, "This email is suspected to be fraudulent. Do not open it."

[1489] Audio analysis of call content

[1490] When a user receives a suspected fraudulent call on their landline or smartphone, the device (smart speaker or smart earphone) records the call in real time. The audio data is sent to a server, which converts it into text. A machine learning algorithm is then used to determine whether the call is suspected to be fraudulent. If a fraudulent call is suspected, the device sends an alert to the user.

[1491] Examples:

[1492] When a user receives a "scam call pretending to be a relative" on their landline, the smart speaker records the call and sends the data to a server. The server converts the call, including keywords such as "money" and "transfer," into text and analyzes it. If the server determines that the call may be fraudulent, the device notifies the user, "This is a potential fraudulent call. Please hang up immediately."

[1493] Mail analysis

[1494] The user takes a photo of a document, such as mail, with the tablet's camera. The device then sends the captured image data to a server. The server then uses image processing technology to extract text from the image and analyzes the extracted text to determine whether there is any suspicion of fraud. If there is any suspicion of fraud, the device sends an alert to the user.

[1495] Examples:

[1496] A user takes a photo of fraudulent mail delivered to their home using the camera on their tablet device. The device sends the image data to a server, which then uses OCR technology to extract text such as "bank" and "transfer" from the document. If the analysis results indicate a suspicion of fraud, the device notifies the user, saying, "This mail is suspected to be fraudulent. Please be careful."

[1497] Sending alerts

[1498] If fraud is suspected, the server generates an alert message and sends it to the device, which then displays the message on the screen of the smart speaker or tablet and, if necessary, notifies the smart earphones via voice notification.

[1499] Examples:

[1500] Based on the result that "fraud is suspected," the server sends an alert message to the device stating "Warning: Possible fraud." The device displays the message on the tablet screen and uses smart earphones to notify the user by voice, "Possible fraud. Please be careful."

[1501] This system can prevent fraud committed through multiple communication methods against users, including the elderly. By using these methods, users are protected from the threat of fraud and can go about their daily lives with peace of mind.

[1502] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1503] Divide the program processing flow into processing steps

[1504] Registration and storage of user information

[1505] Step 1:

[1506] The user opens the Wi-Fi connection settings screen on the tablet device and enters the SSID and password to establish a connection.

[1507] Input: Wi-Fi SSID and password

[1508] Output: Wi-Fi connection establishment status

[1509] Step 2:

[1510] The user enters personal information (name, age, address, etc.) using a tablet device.

[1511] Input: Personal information such as name, age, address, etc.

[1512] Output: Temporarily save the entered personal information

[1513] Step 3:

[1514] The terminal encrypts the information entered and sends it to the server using the SSL / TLS protocol.

[1515] Input: User personal information

[1516] Output: Encrypted transmission of personal information

[1517] Step 4:

[1518] The server decrypts the received information and stores it in a secure database.

[1519] Input: Encrypted personal information

[1520] Output: Personal information stored in the database

[1521] Email Analysis

[1522] Step 1:

[1523] The device periodically checks the user's email account for new emails.

[1524] Input: Email account information

[1525] Output: New emails

[1526] Step 2:

[1527] The device receives new email and retrieves its contents (body, sender, attachments, etc.).

[1528] Input: New email

[1529] Output: Retrieved email content

[1530] Step 3:

[1531] The device encrypts the email content and sends it to the server.

[1532] Input: Email content

[1533] Output: Encrypted email content

[1534] Step 4:

[1535] The server decrypts the received email and analyzes it using machine learning algorithms.

[1536] Input: Encrypted email content

[1537] Output: Analysis results

[1538] Step 5:

[1539] If fraud is suspected, the server generates an alert message and sends it to the terminal.

[1540] Input: Analysis results

[1541] Output: Alert message

[1542] Step 6:

[1543] The device displays an alert to the user.

[1544] Input: Alert message

[1545] Output: User notification

[1546] Audio analysis of call content

[1547] Step 1:

[1548] Users receive suspicious fraudulent calls on their landline or smartphone.

[1549] Input:Incoming call

[1550] Output: Call Start

[1551] Step 2:

[1552] The device records the call in real time.

[1553] Input: Call content

[1554] Output: Recorded audio data

[1555] Step 3:

[1556] The device sends the recorded audio data to the server.

[1557] Input: Recorded audio data

[1558] Output: Transmitted audio data

[1559] Step 4:

[1560] The server converts the received voice data into text.

[1561] Input: Audio data

[1562] Output: The converted text

[1563] Step 5:

[1564] The server uses machine learning algorithms to analyze the text data.

[1565] Input: Translated text

[1566] Output: Analysis results

[1567] Step 6:

[1568] If fraud is suspected, the server generates an alert message and sends it to the terminal.

[1569] Input: Analysis results

[1570] Output: Alert message

[1571] Step 7:

[1572] The device notifies the user of the alert.

[1573] Input: Alert message

[1574] Output: User notification

[1575] Mail analysis

[1576] Step 1:

[1577] The user takes a photo of the mail item using the camera on the tablet device.

[1578] Input: Mail

[1579] Output: Image data

[1580] Step 2:

[1581] The terminal transmits the acquired image data to the server.

[1582] Input: Image data

[1583] Output: Image data sent

[1584] Step 3:

[1585] The server extracts the text from the image.

[1586] Input: Image data

[1587] Output: Extracted text

[1588] Step 4:

[1589] The server analyzes the extracted text using machine learning algorithms.

[1590] Input: Extracted text

[1591] Output: Analysis results

[1592] Step 5:

[1593] If fraud is suspected, the server generates an alert message and sends it to the terminal.

[1594] Input: Analysis results

[1595] Output: Alert message

[1596] Step 6:

[1597] The device notifies the user of the alert.

[1598] Input: Alert message

[1599] Output: User notification

[1600] Sending alerts

[1601] Step 1:

[1602] If fraud is suspected, the server generates an alert message.

[1603] Input: Analysis results

[1604] Output: Alert message

[1605] Step 2:

[1606] The server sends an alert message to the terminal.

[1607] Input: Alert message

[1608] Output: Alert message sent

[1609] Step 3:

[1610] The device will display an alert message on the screen of a smart speaker or tablet, and if necessary, will also notify the smart earphones via voice.

[1611] Input: The alert message sent

[1612] Output: User notification

[1613] This makes it possible to comprehensively detect fraudulent activities carried out through multiple communication means and quickly warn users.

[1614] (Application example 1)

[1615] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1616] In today's world, where users, including the elderly, are at increased risk of falling victim to special frauds, there is a need to prevent fraudulent acts via email, phone calls, and mail. However, it is difficult for users themselves to detect the signs of these frauds, so an efficient fraud prevention system is needed. Furthermore, conventional systems have had issues with the difficulty of real-time monitoring and highly accurate judgment.

[1617] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1618] In this invention, the server includes means for analyzing received emails and determining whether fraud is suspected, means for acquiring and analyzing phone call content and determining whether fraud is suspected, means for scanning documents with a photographing device and determining whether fraud is suspected, means for sending an alert to a user based on the determination result using a generative AI model, and means for generating a prompt message. This allows users to detect the risk of fraud via email, phone calls, or mail in real time and respond quickly.

[1619] "Means for analyzing received e-mails" refers to a function that analyzes the contents of e-mails received by the system and determines whether there is any suspicion of fraud.

[1620] "Means for acquiring and analyzing call content" refers to a function that acquires the content of a user's call, analyzes that content in real time, and determines whether there is any suspicion of fraud.

[1621] "Means for reading documents with a photographic device" refers to a function that allows a user to photograph a specific document with a photographic device such as a camera, and then determine whether or not there is suspicion of fraud based on the image data.

[1622] A "generative AI model" is an algorithmic model trained using machine learning and deep learning techniques that is used to analyze emails, calls, and documents to identify fraud.

[1623] "Means for generating prompts" refers to a function that uses an AI model to create and present appropriate warning messages or alerts to users based on specific conditions or judgment results.

[1624] "Means for sending alerts" refers to a function that sends an alert or message to a user's device to warn them if fraud is suspected.

[1625] As an embodiment of the present invention, a comprehensive system for protecting users, particularly the elderly, from special frauds is provided. The system includes means for automatically analyzing received emails, phone call contents, mail, etc., determining whether there is any suspected fraud, and sending an alert to the user.

[1626] Registration and storage of user information

[1627] During the initial setup, the user uses a tablet or smartphone to enter information (SSID and password) to connect to their home Wi-Fi. They also enter personal information such as their name, age, and address, and the device sends this information to the server. The server receives the information and registers it in a database.

[1628] Email Analysis

[1629] The email analysis method involves the device monitoring the user's email account and sending received emails to a server. The server then uses a generative AI model to analyze the email content and determine whether it is suspected of being fraudulent. If the analysis results indicate a suspicion of fraud, an alert is sent to the device.

[1630] Audio analysis of call content

[1631] When a user receives a suspected fraudulent call on their landline or smartphone, the device (smart speaker or smart earphone) records the call in real time and sends the audio data to a server. The server converts the audio data into text and uses a generative AI model to determine whether the call is suspected to be fraudulent. If fraud is suspected, the device sends the user an alert saying, "This is a potential fraudulent call. Please hang up immediately."

[1632] Mail analysis

[1633] A user takes a photo of a document, such as mail, with the camera on their tablet or smartphone. The device then sends the captured image data to a server. The server extracts text from the image and analyzes it using a generative AI model to determine whether it is suspected of fraud. If fraud is suspected, the device sends the user an alert saying, "This mail is suspected to be fraudulent. Please be careful."

[1634] Sending alerts

[1635] If the server determines that fraud is suspected, it generates a prompt message and sends it to the device, which then displays the message on the screen of the smart speaker or smartphone and, if necessary, sends a voice notification to the smart earphones.

[1636] Hardware and software used

[1637] Hardware: smartphones, tablets, smart speakers, smart earphones, cameras, microphones

[1638] Software: OCR software (e.g., tesseract-ocr), machine learning models (e.g., scikit-learn), speech recognition software (e.g., Google Speech Recognition API), email processing libraries (e.g., python-email, smtplib), call monitoring software (e.g., Twilio API)

[1639] Prompt Sentence Examples

[1640] Use the following as an example prompt for analyzing suspected fraudulent emails:

[1641] "We have come across a potentially fraudulent email. The contents are as follows:

[1642] From: {sender}

[1643] Subject: {subject}

[1644] Contents: {body}

[1645] Please be careful as this appears to be a scam."

[1646] This will enable users to respond to fraudulent activity in real time, improving safety.

[1647] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1648] Step 1: Register and save user information

[1649] During the initial setup, a user uses a tablet or smartphone to enter information (SSID and password) to connect to their home Wi-Fi. They also enter personal information such as their name, age, and address, which the device then sends to the server. The server receives the information and registers it in a database. The input here is the user's Wi-Fi information and personal information, and the output is the user information stored in the database.

[1650] Step 2: Capture and parse emails

[1651] The device monitors the user's email account and sends received emails to a server. The server uses a generative AI model to analyze the email content and determine whether it is suspected of fraud. The input here is the content of the received email, and the output is a judgment result on whether it is suspected of fraud.

[1652] Step 3: Audio analysis of the call

[1653] When a user receives a suspected fraudulent call on their landline or smartphone, the device (smart speaker or smart earphone) records the call in real time and sends the audio data to a server. The server converts the audio data into text and uses a generative AI model to determine whether the call is suspected of fraud. The input here is the recorded call content, and the output is the converted text and a determination of whether the call is suspected of fraud.

[1654] Step 4: Image analysis of mail

[1655] A user takes a photo of a document, such as mail, using a tablet or smartphone camera. The device then sends the captured image data to a server. The server extracts text from the image and uses a generative AI model to analyze the text and determine whether it is suspected of fraud. The input here is the image of the mail, and the output is the extracted text and a determination of whether it is suspected of fraud.

[1656] Step 5: Sending an alert

[1657] If the server determines that fraud is suspected, it generates a prompt and an alert message and sends it to the device. The device then displays the warning message on the screen of the smart speaker or smartphone, and if necessary, notifies the smart earphones by voice. The input here is the judgment result regarding the suspected fraud, and the output is the generated prompt and alert message.

[1658] Through these steps, users can detect fraud risks in real time and respond quickly.

[1659] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1660] As an embodiment for carrying out the present invention, various means of a system (hereinafter referred to as an improved system) that combines a special fraud prevention system with an emotion engine will be described.

[1661] Registration and storage of user information

[1662] During the initial setup, the user uses the tablet device to enter the SSID and password to connect to their home Wi-Fi. After that, they enter personal information such as their name, age, and address, and the device sends this information to the server. The server receives the information and registers it in a database.

[1663] Email analysis and emotion recognition

[1664] The device monitors the user's email account and sends received emails to a server. The server analyzes the email content and determines whether it is suspected of being fraudulent. It also recognizes the user's emotions based on the email content. If the analysis results indicate a suspicion of fraud, it sends an alert to the device.

[1665] Voice analysis and emotion recognition of call content

[1666] When a user receives a suspected fraudulent call on their landline or smartphone, the smart speaker or earphone device records the call in real time and sends the audio data to a server. The server converts the audio data into text and then analyzes the text to determine whether the call is suspected of fraud. The system also recognizes the user's emotions from the call content and sends an alert if there is a change in emotion.

[1667] Mail analysis and emotion recognition

[1668] The user takes a photo of a document, such as mail, using the tablet's camera. The device then sends the captured image data to a server. The server extracts text from the image and analyzes it to determine whether it is suspected of fraud. In addition, the system recognizes the user's emotions based on the document's content. If it determines that the document is suspected of fraud, the device sends an alert to the user.

[1669] Sending alerts

[1670] If fraud is suspected, the server generates an alert message and sends it to the device. The device then displays the message on the screen of the smart speaker or tablet, and if necessary, sends a voice notification to the smart earphones. The emotion engine also notifies the user of the analysis results and provides appropriate advice based on the user's psychological state.

[1671] Specific examples

[1672] Specific examples of mail analysis

[1673] 1. The user takes a photo of mail delivered to their home with their camera.

[1674] 2. The device sends the image data to the server.

[1675] 3. The server extracts text from the image and analyzes the text using machine learning algorithms.

[1676] 4. The server further recognizes the user's emotions based on the document content.

[1677] 5. If it determines that there is a suspicion of fraud, the device will send the user an alert saying, "This mail is suspected to be fraudulent. Please be careful," and will also send an emotional alert saying, "This mail conveys a sense of anxiety. Please ignore it."

[1678] Specific examples of call analysis

[1679] 1. A user receives a suspected fraudulent call on their landline.

[1680] 2. The smart speaker device records the call and sends the data to the server.

[1681] 3. The server converts the speech into text and analyzes it.

[1682] 4. The server recognizes the user's emotions from the content of the call and sends an alert if there is a change in emotions.

[1683] 5. If fraud is suspected, the device will send the user an alert saying, "This is a potential fraudulent call. Please hang up immediately," and will also provide an emotional alert saying, "You are feeling surprised and anxious. Please hang up."

[1684] As a result, the improved system can prevent the threat of special fraud and ensure safety for users, including the elderly. In addition, it can recognize the user's emotional state in real time and provide psychological support.

[1685] The processing flow will be explained below.

[1686] Email analysis and emotion recognition

[1687] Step 1:

[1688] The user receives an email.

[1689] Step 2:

[1690] The terminal acquires the contents of the received email and sends them to the server.

[1691] Step 3:

[1692] The server analyzes the content of the email to determine whether it is suspected of being fraudulent.

[1693] Step 4:

[1694] The server recognizes the user's emotions based on the email content.

[1695] Step 5:

[1696] The server sends the analysis results and emotion recognition results to the device.

[1697] Step 6:

[1698] If fraud is suspected, the device will display or notify the user of an alert and an emotion alert.

[1699] Voice analysis and emotion recognition of call content

[1700] Step 1:

[1701] A user initiates a call on a desk phone or smartphone.

[1702] Step 2:

[1703] The device, a smart speaker or smart earphones, records the call in real time.

[1704] Step 3:

[1705] The device sends the recorded audio data to the server.

[1706] Step 4:

[1707] The server converts the audio data into text.

[1708] Step 5:

[1709] The server analyzes the converted text to determine whether it is suspected of fraud.

[1710] Step 6:

[1711] The server recognizes the user's emotions based on the contents of the call.

[1712] Step 7:

[1713] The server sends the analysis results and emotion recognition results to the device.

[1714] Step 8:

[1715] If fraud is suspected, the device will display an alert and emotion alert to the user or notify them audibly during the call.

[1716] Mail analysis and emotion recognition

[1717] Step 1:

[1718] The user takes a photo of the mail that arrives at their home using the camera on their tablet device.

[1719] Step 2:

[1720] The terminal transmits the captured image data to the server.

[1721] Step 3:

[1722] The server extracts the text from the image.

[1723] Step 4:

[1724] The server analyzes the extracted text to determine whether it is suspected of fraud.

[1725] Step 5:

[1726] The server recognizes the user's emotions based on the document content.

[1727] Step 6:

[1728] The server sends the analysis results and emotion recognition results to the device.

[1729] Step 7:

[1730] If fraud is suspected, the device will display or notify the user of an alert and an emotion alert.

[1731] account settings

[1732] Step 1:

[1733] The user opens the initial setup screen using the tablet device.

[1734] Step 2:

[1735] The user enters the Wi-Fi SSID and password.

[1736] Step 3:

[1737] The device attempts to connect to Wi-Fi and displays the results to the user.

[1738] Step 4:

[1739] The user enters personal information such as name, age, and address.

[1740] Step 5:

[1741] The terminal transmits the input information to the server.

[1742] Step 6:

[1743] The server receives the information and stores it in a database.

[1744] Step 7:

[1745] The server sends a confirmation message to the terminal that the save has been completed, and the terminal notifies the user.

[1746] Through these processing steps, the improved system can ensure the safety of users and prevent them from committing fraud. Furthermore, by utilizing the emotion engine, it is possible to determine the user's emotional state in real time and provide psychological support.

[1747] Example 2

[1748] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1749] In modern society, special frauds are becoming increasingly sophisticated and their methods are diversifying. Fraud targeting the elderly in particular has become a major social problem, and conventional fraud prevention systems are often unable to keep up with new methods. Furthermore, the lack of appropriate countermeasures and support that take into account the user's emotional state makes it difficult to prevent damage before it occurs. To solve this problem, it is necessary to provide a system that can effectively analyze communication methods (email, phone calls, documents) that are suspected of being fraudulent, and that can recognize and respond to the user's emotional state in real time.

[1750] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1751] In this invention, the server includes means for analyzing received electronic messages and determining whether fraud is suspected, means for acquiring and analyzing the contents of phone calls and determining whether fraud is suspected, means for photographing and reading documents and determining whether fraud is suspected, means for analyzing the emotional state of the user, and means for sending an alert to the user based on the determination result and the emotional state. This makes it possible to deal with a variety of fraudulent methods, provide appropriate support that takes into account the psychological state of the user, and prevent fraud damage before it occurs.

[1752] "Means for analyzing received electronic messages to determine whether they are suspected of fraud" refers to a combination of software and hardware that processes the content of emails and messages received by a user and automatically detects possible fraudulent activity.

[1753] "Means for acquiring and analyzing call content to determine whether fraud is suspected" refers to a combination of software and hardware that records the user's call content and analyzes the audio data to identify possible fraud.

[1754] "Means for photographing and reading documents to determine whether they are suspected of fraud" refers to a combination of software and hardware that converts the contents of a document photographed by a user with a camera into text data using technologies such as optical character recognition, and then analyzes the text data to determine whether it is suspected of fraud.

[1755] The "means for analyzing a user's emotional state" is a combination of software and hardware for analyzing a user's emotional state based on the contents of electronic messages, conversations and documents, and evaluating the results.

[1756] The "means for sending an alert to the user based on the judgment result and emotional state" refers to a combination of software and hardware that, when a suspected fraud is judged to be present, generates a warning message that takes into account the judgment result and the user's emotional state and notifies the user.

[1757] "Means for registering and storing user information" refers to a combination of software and hardware for registering users' personal information in a database and using or referencing it when necessary.

[1758] "Means for monitoring a user's voice communications in real time and sending an alert if the communications contain suspected fraudulent content" refers to a combination of software and hardware that monitors a user's calls in real time and immediately issues an alert if suspected fraudulent content is detected.

[1759] As an embodiment of the present invention, the detailed configuration and operation of a system (hereinafter referred to as "this system") that combines a special fraud prevention system with an emotion recognition engine will be described.

[1760] Registration and storage of user information

[1761] The user uses the tablet device for initial setup and enters the SSID and password to connect to their home Wi-Fi. Next, they enter personal information such as their name, age, and address into the device. This information is sent from the device to the server. The server registers the received information in a database, which allows the user's basic information to be safely stored.

[1762] Analysis of electronic messages and emotion recognition

[1763] The device periodically monitors the user's email account and receives new emails. The received emails are immediately sent to the server. The server uses a text analysis engine to analyze the email content and determine whether it is suspected of being fraudulent. The server also uses an emotion recognition engine to extract the user's emotions from the email content. If it determines that the email is suspected of being fraudulent, it sends an alert message to the device and displays a warning to the user.

[1764] Voice analysis and emotion recognition of call content

[1765] When a user receives a suspected fraudulent call on their landline or smartphone, the smart speaker or earphones that are part of the device record the call in real time. The recorded data is then immediately sent to a server. The server uses a speech recognition engine to convert the voice data into text and analyzes the text to determine whether the call is suspected of fraud. The server also recognizes the user's emotions from the call content and sends an alert if a change in emotion is detected.

[1766] Document Analysis and Emotion Recognition

[1767] The user takes a photo of a document, such as mail, with the tablet's camera. The device then sends the captured image data to the server. The server uses an image analysis engine to extract text from the image and analyzes that text to determine whether it is suspected of fraud. The server then recognizes the user's emotions based on the document's content. If it determines that the document is suspected of fraud, the device sends an alert to the user.

[1768] Sending alerts

[1769] If a suspected fraud attempt is detected, the server generates an alert message and sends it to the device, which then displays the message on the screen of the smart speaker or tablet and, if necessary, also notifies the smart earphones via voice notification, allowing the user to immediately become aware of the possibility of fraud.

[1770] Specific examples

[1771] Specific examples of mail analysis

[1772] 1. The user takes a photo of the mail that arrives at their home with their camera.

[1773] 2. The device sends the image data to the server.

[1774] 3. The server extracts text from the image and analyzes the text using machine learning algorithms.

[1775] 4. The server further recognizes the user's emotions based on the document content.

[1776] 5. If it determines that there is a suspicion of fraud, the device will send the user an alert saying, "This mail is suspected to be fraudulent. Please be careful," and will also send an emotional alert saying, "This mail conveys a sense of anxiety. Please ignore it."

[1777] Specific examples of call analysis

[1778] 1. A user receives a suspected fraudulent call on their landline.

[1779] 2. The smart speaker device records the call and sends the data to the server.

[1780] 3. The server converts the speech into text and analyzes it.

[1781] 4. The server recognizes the user's emotions from the content of the call and sends an alert if there is a change in emotion.

[1782] 5. If fraud is suspected, the device will send the user an alert saying, "This is a potential fraudulent call. Please hang up immediately," and will also provide an emotional alert saying, "You are feeling surprised and anxious. Please hang up."

[1783] In this way, this system can prevent the threat of special fraud and ensure the safety of users, including the elderly. In addition, it can recognize the user's emotional state in real time and provide psychological support.

[1784] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1785] Registration and storage of user information

[1786] Processing Steps

[1787] Step 1:

[1788] The user opens the settings screen on the tablet device and enters the SSID and password to connect to their home Wi-Fi.

[1789] Input: SSID, password

[1790] Output: Wi-Fi connection complete message

[1791] Specific operation: The user enters the SSID and password in the specified fields and presses the "Connect" button. The device receives this and attempts to connect to Wi-Fi. If the connection is successful, a connection completion message is displayed.

[1792] Step 2:

[1793] The user enters personal information such as name, age, and address into the terminal.

[1794] Input: Personal information such as name, age, address, etc.

[1795] Output: Input confirmation screen

[1796] Specific operation: The user enters personal information into each field and presses the "Submit" button. The device retrieves this information and displays a confirmation screen.

[1797] Step 3:

[1798] The terminal transmits the entered personal information to the server.

[1799] Input: Personal information entered on the device

[1800] Output: Message that data has been sent to the server

[1801] Specific operation: The terminal packetizes the entered personal information and sends it to the server using a secure communication protocol (e.g., HTTPS).

[1802] Step 4:

[1803] The server registers the received personal information in a database.

[1804] Input: Personal information sent from the device

[1805] Output: Database registration completion message

[1806] Specific operation: The server analyzes the received data and stores it in the database appropriately. Once the data has been stored, it returns a registration completion message to the device.

[1807] Analysis of electronic messages and emotion recognition

[1808] Processing Steps

[1809] Step 1:

[1810] The device periodically monitors the user's email account and receives new emails.

[1811] Input: Email login details

[1812] Output: New email information

[1813] Specific operation: The device sends a query to the mail server at the specified interval to check for new mail. If new mail is found, the device retrieves its contents.

[1814] Step 2:

[1815] The device sends the contents of the new email to the server.

[1816] Input: New email content

[1817] Output: Message that email has been sent to the server

[1818] Specific operation: The device obtains the body and header information of the new email and sends it to the server.

[1819] Step 3:

[1820] The server uses a text analysis engine to analyze the email content and determine whether it is suspected of being fraudulent.

[1821] Input: Email text data

[1822] Output: Fraud detection result

[1823] What it does: The server uses a text analysis engine (e.g., an NLP model) to analyze the content of the email and determine whether it is fraudulent. The result is stored as a flag.

[1824] Step 4:

[1825] The server uses an emotion recognition engine to extract the user's emotion from the content of the email.

[1826] Input: Email text data

[1827] Output: Emotion recognition result

[1828] Specific operation: The server uses an emotion recognition engine (e.g., emotion analysis API) to extract the user's emotion from the email body and saves the result as a flag.

[1829] Step 5:

[1830] If fraud is suspected, an alert message will be sent to the terminal and a warning will be displayed to the user.

[1831] Input: Fraud detection result, emotion recognition result

[1832] Output: Alert message screen

[1833] Specific operation: If the fraud flag is set, the server generates an alert message and sends it to the terminal. The terminal receives it and displays the warning message to the user.

[1834] Voice analysis and emotion recognition of call content

[1835] Processing Steps

[1836] Step 1:

[1837] Users receive suspicious fraudulent calls on their landline or smartphone.

[1838] Input: Incoming call notification

[1839] Output: Call start signal

[1840] Specific operation: The user receives an incoming call and answers the phone. The device detects this and transitions to a call state.

[1841] Step 2:

[1842] Smart speakers and smart earphones that are part of the device record the contents of calls in real time.

[1843] Input: Call audio data

[1844] Output: Recording data

[1845] Specific operation: The terminal starts recording as soon as the call starts and stores the audio data in a buffer.

[1846] Step 3:

[1847] The recording data is immediately sent to the server.

[1848] Input: Recording data

[1849] Output: Message that data has been sent to the server

[1850] What happens: The device sends the recording data to the server using a secure protocol (e.g., HTTPS).

[1851] Step 4:

[1852] The server uses a speech recognition engine to convert the voice data into text.

[1853] Input: Audio data

[1854] Output: Text data

[1855] Specific operation: The server uses a speech recognition engine (e.g., Speech-to-Text API) to convert the voice data into text.

[1856] Step 5:

[1857] The server analyzes the text to determine if it is suspected of fraud.

[1858] Input: Converted text data

[1859] Output: Fraud detection result

[1860] What it does: The server uses a text analysis engine to determine the likelihood of fraud, and stores the result as a flag.

[1861] Step 6:

[1862] The server recognizes the user's emotions from the content of the call and sends an alert if there is a change in emotion.

[1863] Input: Text data

[1864] Output: Emotion recognition result

[1865] Specific operation: The server uses an emotion recognition engine to extract the user's emotion from the text data and saves the result as a flag.

[1866] Step 7:

[1867] If fraud is suspected, an alert message will be sent to the terminal and a warning will be displayed to the user.

[1868] Input: Fraud detection result, emotion recognition result

[1869] Output: Alert message screen

[1870] Specific operation: If the fraud flag is set, the server generates an alert message and sends it to the terminal. The terminal receives it and displays the warning message to the user.

[1871] Document Analysis and Emotion Recognition

[1872] Processing Steps

[1873] Step 1:

[1874] The user takes a photo of the mail item using the camera on the tablet device.

[1875] Input: Image data of mail

[1876] Output: Shooting completion message

[1877] Specific operation: The user takes a photo of the mail using the camera function of the tablet device. The image data is stored on the device.

[1878] Step 2:

[1879] The terminal transmits the image data to the server.

[1880] Input: Image data of mail

[1881] Output: Message that data has been sent to the server

[1882] Specific operation: The device sends the acquired image data to the server using a secure protocol (e.g., HTTPS).

[1883] Step 3:

[1884] The server uses an image analysis engine to extract text from the image.

[1885] Input: Image data

[1886] Output: Extracted text data

[1887] Specific operation: The server uses an image analysis engine (e.g., OCR API) to extract text data from the image.

[1888] Step 4:

[1889] The server analyzes the text to determine if it is suspected of fraud.

[1890] Input: Extracted text data

[1891] Output: Fraud detection result

[1892] What it does: The server uses a text analysis engine to determine the likelihood of fraud, and stores the result as a flag.

[1893] Step 5:

[1894] The server recognizes the user's emotions based on the document content.

[1895] Input: Extracted text data

[1896] Output: Emotion recognition result

[1897] Specific operation: The server uses an emotion recognition engine to extract the user's emotion from the document content, and the result is saved as a flag.

[1898] Step 6:

[1899] If fraud is suspected, an alert message will be sent to the terminal and a warning will be displayed to the user.

[1900] Input: Fraud detection result, emotion recognition result

[1901] Output: Alert message screen

[1902] Specific operation: If the fraud flag is set, the server generates an alert message and sends it to the terminal. The terminal receives it and displays a warning message to the user.

[1903] Sending alerts

[1904] Processing Steps

[1905] Step 1:

[1906] The server detects suspected fraud.

[1907] Input: Fraud detection result, emotion recognition result

[1908] Output: Alert message

[1909] Specific operation: The server checks the fraud determination flag and emotion recognition flag and generates an alert message if necessary.

[1910] Step 2:

[1911] The server generates an alert message and sends it to the terminal.

[1912] Input: Fraud detection result, emotion recognition result

[1913] Output: The generated alert message

[1914] Specific operation: The server generates an alert message and sends it to the terminal in an appropriate format.

[1915] Step 3:

[1916] The device displays or notifies you of an alert message.

[1917] Input: The generated alert message

[1918] Output: The displayed warning message screen

[1919] Specific operation: When the terminal receives an alert message, it displays a visual or audio warning message to the user.

[1920] Through the above processing steps, the system can provide high detection accuracy and rapid response to a variety of fraudulent methods, while also providing appropriate support based on the user's emotional state.

[1921] (Application example 2)

[1922] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1923] In recent years, there has been an increase in special frauds targeting the elderly, and many of these are committed via email, telephone, or mail. This has led to a growing need for systems that can not only detect fraud but also monitor users' emotions in real time and provide appropriate responses based on their psychological state. While conventional systems specialize in detecting fraud, they do not provide support that takes users' emotions into consideration, resulting in a lack of psychological support.

[1924] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1925] In this invention, the server includes means for analyzing received electronic documents and determining whether fraud is suspected, means for acquiring and analyzing the contents of phone calls and determining whether fraud is suspected, means for scanning documents with an imaging device and determining whether fraud is suspected, means for recognizing the user's emotions using an emotion recognition engine when determining whether fraud is suspected, and providing appropriate advice on how to respond if there is a possibility of fraud, and means for sending a warning to the user based on the determination result. This makes it possible not only to respond immediately to information that is suspected of being fraud, but also to provide psychological support that takes into account the user's emotional state.

[1926] An "electronic document" is a document stored in digital format, including email, PDF, Word files, etc.

[1927] "Call content" refers to information exchanged through voice communication, such as the content of a conversation conducted via telephone or internet telephone.

[1928] "Photographing device" refers to equipment for taking still or moving images, and includes devices with camera functions.

[1929] An "emotion recognition engine" is a combination of software and hardware for identifying and analyzing human emotional states.

[1930] A "warning" is a message or notification that warns the user of a caution or danger based on a specific condition.

[1931] "User Information" refers to personal data relating to users of the system, including name, age, address, contact details, etc.

[1932] "Analysis" refers to the process of examining a set of data or information in detail to understand its content, trends, and relationships.

[1933] "Judgment" is the act of determining an outcome based on collected data and information according to specific conditions and criteria.

[1934] System Overview

[1935] This system is a security system that combines fraud prevention and user emotion recognition. The system analyzes the contents of received electronic documents, phone calls, and documents captured using a camera, and issues a warning if fraud is suspected. Furthermore, the system uses an emotion recognition engine to analyze the user's emotional state and provide appropriate responses.

[1936] Hardware and software used

[1937] Hardware:

[1938] Smartphone

[1939] camera

[1940] software:

[1941] Server side: Django

[1942] Emotion Recognition API:Azure Cognitive Services Emotion API

[1943] NLP libraries: spaCy, nltk

[1944] Deep Learning: TensorFlow, PyTorch

[1945] Speech Recognition: Google Speech-to-Text API

[1946] OCR library: Tesseract

[1947] Program processing

[1948] 1. Electronic Document Analysis:

[1949] The server periodically retrieves emails received via the API and performs text analysis using an NLP library (spaCy or nltk).

[1950] It analyzes text structure and keywords, and if there is a possibility of fraud, it uses the Azure Cognitive Services Emotion API to analyze the user's emotions.

[1951] If fraud is suspected, a warning message will be generated and sent to the user's smartphone.

[1952] 2. Call analysis:

[1953] It monitors smartphone calls in real time and converts audio data into text using the Google Speech-to-Text API.

[1954] Use NLP libraries to analyze text and determine potential fraud.

[1955] The Azure Cognitive Services Emotion API analyzes the user's emotions, and if anxiety or surprise is detected, a warning message and emotion-based advice is sent to the user.

[1956] 3. Document Analysis:

[1957] A photograph of a document such as mail is taken using a photographing device (smartphone camera), and the image data is sent to the server.

[1958] The server uses an OCR library (Tesseract) to extract text from image data and performs text analysis using an NLP library.

[1959] If fraud is suspected, emotion recognition is performed, a message advising appropriate action is generated, and a warning is sent to the user's smartphone.

[1960] Specific examples

[1961] Specific examples of email analysis

[1962] Process flow: Retrieve email from SMTP server → Analyze text using spaCy → Analyze emotions using Azure Cognitive Services Emotion API → Warn user if fraud is suspected

[1963] Example alerts: "This email may be fraudulent. Please be cautious.", "This email makes me feel uneasy. Please ignore it."

[1964] Prompt Sentence Examples

[1965] "Please analyze the content of the email below to determine the likelihood of a scam and the associated emotions.

[1966] The email reads: "Dear customer, you have been selected for our special campaign. Please update your account information to receive your prize."

[1967] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1968] Step 1: Obtaining electronic documents

[1969] Description: The server periodically retrieves the user's email via API.

[1970] Input: User's email account information

[1971] Output: Retrieved email data

[1972] What happens: The server fetches emails through the SMTP server's API and retrieves new emails in the inbox.

[1973] Step 2: Text analysis

[1974] Description: The server performs text analysis on the body of the retrieved email.

[1975] Input: Email data

[1976] Output: Analysis results (suspected fraud)

[1977] What it does: The server uses an NLP library (e.g. spaCy) to tokenize the email body, extract keywords, and perform contextual analysis to assess the likelihood of fraud.

[1978] Step 3: Sentiment Analysis

[1979] Description: The server analyzes the user's emotional state based on the content of the email.

[1980] Input: Email data, NLP analysis results

[1981] Output: Emotion analysis results

[1982] How it works: The server uses the Azure Cognitive Services Emotion API to estimate the emotions the user may be feeling from the content of the email.

[1983] Step 4: Generate warning messages

[1984] Description: If fraud is suspected, the server generates a warning message.

[1985] Input: NLP analysis results, sentiment analysis results

[1986] Output: Warning message

[1987] Specific behavior: The server generates a warning message based on the analysis results, for example, "This email may be fraudulent. Please be careful."

[1988] Step 5: Sending an alert

[1989] Description: The server sends the generated warning message to the user's terminal.

[1990] Input: warning message

[1991] Output: A warning displayed on the user's terminal

[1992] Specific operation: The server sends a warning message to the user's smartphone via push notification.

[1993] Step 6: Capture the call

[1994] Description: The device receives the user's call content in real time.

[1995] Input: Call audio data

[1996] Output: Recorded audio data

[1997] How it works: The smartphone uses its built-in microphone to record the conversation and sends it to the server as audio data.

[1998] Step 7: Speech to Text

[1999] Description: The server converts the call audio data into text.

[2000] Input: Captured audio data

[2001] Output: Text data

[2002] Specific operation: The server uses the Google Speech-to-Text API to convert the captured voice data into text.

[2003] Step 8: Call Analysis

[2004] Description: The server analyzes the text of the call to determine if it is suspected of fraud.

[2005] Input: Text data

[2006] Output: Analysis results (suspected fraud)

[2007] What it does: The server uses an NLP library to tokenize the call, extract keywords and perform contextual analysis to assess the likelihood of fraud.

[2008] Step 9: Sentiment Analysis

[2009] Description: The server analyzes the user's emotional state from the content of the call.

[2010] Input: Text data, NLP analysis results

[2011] Output: Emotion analysis results

[2012] Specific operation: The server uses the Azure Cognitive Services Emotion API to estimate the user's emotions from the content of the call.

[2013] Step 10: Generate warning messages

[2014] Description: If fraud is suspected or there is a change in sentiment, the server generates a warning message.

[2015] Input: NLP analysis results, sentiment analysis results

[2016] Output: Warning message

[2017] Specific behavior: Based on the analysis results, the server generates a warning message such as "This call may be fraudulent. Please hang up immediately."

[2018] Step 11: Sending an alert

[2019] Description: The server sends the generated warning message to the user's terminal.

[2020] Input: warning message

[2021] Output: A warning displayed on the user's terminal

[2022] Specific operation: The server sends a warning message to the user's smartphone via push notification.

[2023] Step 12: Photograph the document

[2024] Description: A user takes a picture of a document, such as mail, with a camera.

[2025] Input: Image data

[2026] Output: Photographed document image

[2027] Specific action: The user uses the smartphone camera to take a picture of the mail.

[2028] Step 13: Extracting text from images

[2029] Description: The server extracts text from the captured image.

[2030] Input: Image data

[2031] Output: Extracted text data

[2032] Specific operation: The server uses an OCR library (Tesseract) to extract text from the image.

[2033] Step 14: Document Content Analysis

[2034] Description: The server analyzes the text data to determine if it is suspected of fraud.

[2035] Input: Text data

[2036] Output: Analysis results (suspected fraud)

[2037] How it works: The server uses an NLP library to tokenize the document content, extract keywords and perform contextual analysis to assess the likelihood of fraud.

[2038] Step 15: Sentiment Analysis

[2039] Description: The server analyzes the user's emotional state from the document content.

[2040] Input: Text data, NLP analysis results

[2041] Output: Emotion analysis results

[2042] Specific operation: The server uses the Azure Cognitive Services Emotion API to estimate the user's emotions from the document content.

[2043] Step 16: Generate warning messages

[2044] Description: If fraud is suspected, the server generates a warning message.

[2045] Input: NLP analysis results, sentiment analysis results

[2046] Output: Warning message

[2047] Specific operation: Based on the analysis results, the server creates a message such as "This mail is suspected to be fraudulent. Please be careful."

[2048] Step 17: Sending Alerts

[2049] Description: The server sends the generated warning message to the user's terminal.

[2050] Input: warning message

[2051] Output: A warning displayed on the user's terminal

[2052] Specific operation: The server sends a warning message to the user's smartphone via push notification.

[2053] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[2054] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2055] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[2056] [Fourth embodiment]

[2057] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[2058] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[2059] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[2060] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[2061] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[2062] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[2063] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[2064] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[2065] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[2066] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[2067] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[2068] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[2069] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2070] As an embodiment of the present invention, various means constituting a special fraud prevention system (hereinafter referred to as the present system) will be described.

[2071] Registration and storage of user information

[2072] During the initial setup, the user uses the tablet device to enter the SSID and password to connect to their home Wi-Fi. After that, they enter personal information such as their name, age, and address, and the device sends this information to the server. The server receives the information and registers it in a database.

[2073] Email Analysis

[2074] The device monitors the user's email account and sends received emails to a server, which analyzes the email content and uses machine learning algorithms to determine whether the email is suspected of being fraudulent. If the analysis results indicate a suspected fraud, an alert is sent to the device.

[2075] Audio analysis of call content

[2076] When a user receives a suspected fraudulent call on their landline or smartphone, the device (smart speaker or smart earphone) records the call in real time and sends the audio data to a server. The server converts the audio data into text and determines whether it is suspected to be fraudulent. If it is, the device sends an alert to the user.

[2077] Mail analysis

[2078] The user takes a photo of a document, such as mail, with the tablet's camera. The device then sends the captured image data to a server. The server extracts text from the image and analyzes it to determine whether it is suspected of fraud. If it is suspected of fraud, the device sends an alert to the user.

[2079] Sending alerts

[2080] If fraud is suspected, the server generates an alert message and sends it to the device, which then displays the message on the screen of the smart speaker or tablet and, if necessary, notifies the smart earphones via voice notification.

[2081] Specific examples

[2082] Specific examples of mail analysis

[2083] 1. The user takes a photo of mail delivered to their home with their camera.

[2084] 2. The device sends the image data to the server.

[2085] 3. The server extracts text from the image and analyzes the text using machine learning algorithms.

[2086] 4. If it is determined that there is a suspicion of fraud, the device will send an alert to the user saying, "This mail is suspected to be fraudulent. Please be careful."

[2087] Specific examples of call analysis

[2088] 1. A user receives a suspected fraudulent call on their landline.

[2089] 2. The smart speaker device records the call and sends the data to the server.

[2090] 3. The server converts the speech into text and analyzes it.

[2091] 4. If fraud is suspected, the device will send an alert to the user saying, "This is a potential fraudulent call. Please hang up immediately."

[2092] In this way, the system can prevent the threat of special frauds and ensure safety for users, including the elderly.

[2093] The processing flow will be explained below.

[2094] Email Analysis

[2095] Step 1:

[2096] The user receives an email.

[2097] Step 2:

[2098] The terminal acquires the contents of the received email and sends them to the server.

[2099] Step 3:

[2100] The server analyzes the content of the email to determine whether it is suspected of being fraudulent.

[2101] Step 4:

[2102] The server sends the analysis results to the device.

[2103] Step 5:

[2104] If fraud is suspected, the device will display an alert or notify the user.

[2105] Audio analysis of call content

[2106] Step 1:

[2107] A user initiates a call on a desk phone or smartphone.

[2108] Step 2:

[2109] The device, a smart speaker or smart earphones, records the call in real time.

[2110] Step 3:

[2111] The device sends the recorded audio data to the server.

[2112] Step 4:

[2113] The server converts the audio data into text.

[2114] Step 5:

[2115] The server analyzes the converted text to determine whether it is suspected of fraud.

[2116] Step 6:

[2117] If fraud is suspected, the device will display an alert to the user or notify them audibly during the call.

[2118] Mail analysis

[2119] Step 1:

[2120] The user takes a photo of the mail that arrives at their home using the camera on their tablet device.

[2121] Step 2:

[2122] The terminal transmits the captured image data to the server.

[2123] Step 3:

[2124] The server extracts the text from the image.

[2125] Step 4:

[2126] The server analyzes the extracted text to determine whether it is suspected of fraud.

[2127] Step 5:

[2128] If fraud is suspected, the device will display an alert or notify the user.

[2129] account settings

[2130] Step 1:

[2131] The user opens the initial setup screen using the tablet device.

[2132] Step 2:

[2133] The user enters the Wi-Fi SSID and password.

[2134] Step 3:

[2135] The device attempts to connect to Wi-Fi and displays the results to the user.

[2136] Step 4:

[2137] The user enters personal information such as name, age, and address.

[2138] Step 5:

[2139] The terminal transmits the input information to the server.

[2140] Step 6:

[2141] The server receives the information and stores it in a database.

[2142] Step 7:

[2143] The server sends a confirmation message to the terminal that the save has been completed, and the terminal notifies the user.

[2144] Through these processing steps, the system is designed to ensure the safety of users and prevent special frauds from occurring.

[2145] Example 1

[2146] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2147] With the development of modern communication methods, frauds using email, telephone, and mail are on the rise. These frauds have become a problem that causes significant economic losses and psychological burdens to many people, including the elderly. Existing countermeasures have limitations, and more effective and comprehensive measures are needed.

[2148] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[2149] In this invention, the server includes means for analyzing received emails and determining whether they are suspected of fraud using a machine learning algorithm, means for recording phone calls in real time, converting the audio data into text, and determining whether they are suspected of fraud, and means for scanning documents with a camera, extracting text from the image data, and determining whether they are suspected of fraud. This makes it possible to comprehensively detect fraudulent activities committed through multiple communication means and quickly warn users.

[2150] A "machine learning algorithm" is a computational method that allows a computer to learn patterns from data and make predictions or classifications.

[2151] "Voice data" refers to data that represents call content and voice information in digital format.

[2152] "Convert to text" is the process of converting non-text data, such as audio or images, into written information.

[2153] "Document" refers to information recorded in writing, including, for example, mail, letters, notices, etc.

[2154] A "camera" is a device that takes an image of an object and stores it as digital data.

[2155] "Image data" refers to data in which images taken with a camera or the like are stored in digital format.

[2156] "Text extraction" is the process of extracting textual information from image or audio data.

[2157] A "server" is a computer system that stores and processes data and provides services to clients over a network.

[2158] An "alert" is a notification that alerts or warns the user, and is provided as a voice or text message.

[2159] "User" refers to anyone who uses this system, and is primarily intended to prevent fraud.

[2160] As an embodiment of the present invention, various means of the special fraud prevention system (hereinafter referred to as the "system") will be specifically described. The system has the following main functions: registration and storage of user information, analysis of e-mails, analysis of voice of phone calls, analysis of mail, and sending of alerts.

[2161] Registration and storage of user information

[2162] The user performs initial setup using the tablet device. First, they enter the SSID and password to connect to their home Wi-Fi. Next, they enter personal information such as their name, age, and address. The entered information is sent from the device to a server, and the server registers the received information in a database. The software used in this process must support a basic user input form and SSL / TLS encrypted communication.

[2163] Email Analysis

[2164] The device monitors the user's email account and sends received emails to a server. The server uses machine learning algorithms to analyze the email content. This analysis process includes detecting fraud-related keywords and verifying sender information. If the analysis results indicate a suspected fraud, the server generates an alert and sends it to the device. The device then displays the alert to the user.

[2165] Examples:

[2166] When a user receives a fraudulent email from the email address "example@example.com," the device sends the contents of the email to the server. The server analyzes the email using a machine learning algorithm that includes the word "fraud." If the server determines that the email is suspected to be fraudulent, it displays a message on the device saying, "This email is suspected to be fraudulent. Do not open it."

[2167] Audio analysis of call content

[2168] When a user receives a suspected fraudulent call on their landline or smartphone, the device (smart speaker or smart earphone) records the call in real time. The audio data is sent to a server, which converts it into text. A machine learning algorithm is then used to determine whether the call is suspected to be fraudulent. If a fraudulent call is suspected, the device sends an alert to the user.

[2169] Examples:

[2170] When a user receives a "scam call pretending to be a relative" on their landline, the smart speaker records the call and sends the data to a server. The server converts the call, including keywords such as "money" and "transfer," into text and analyzes it. If the server determines that the call may be fraudulent, the device notifies the user, "This is a potential fraudulent call. Please hang up immediately."

[2171] Mail analysis

[2172] The user takes a photo of a document, such as mail, with the tablet's camera. The device then sends the captured image data to a server. The server then uses image processing technology to extract text from the image and analyzes the extracted text to determine whether there is any suspicion of fraud. If there is any suspicion of fraud, the device sends an alert to the user.

[2173] Examples:

[2174] A user takes a photo of fraudulent mail delivered to their home using the camera on their tablet device. The device sends the image data to a server, which then uses OCR technology to extract text such as "bank" and "transfer" from the document. If the analysis results indicate a suspicion of fraud, the device notifies the user, saying, "This mail is suspected to be fraudulent. Please be careful."

[2175] Sending alerts

[2176] If fraud is suspected, the server generates an alert message and sends it to the device, which then displays the message on the screen of the smart speaker or tablet and, if necessary, notifies the smart earphones via voice notification.

[2177] Examples:

[2178] Based on the result that "fraud is suspected," the server sends an alert message to the device stating "Warning: Possible fraud." The device displays the message on the tablet screen and uses smart earphones to notify the user by voice, "Possible fraud. Please be careful."

[2179] This system can prevent fraud committed through multiple communication methods against users, including the elderly. By using these methods, users are protected from the threat of fraud and can go about their daily lives with peace of mind.

[2180] The flow of the identification process in the first embodiment will be described with reference to FIG.

[2181] Divide the program processing flow into processing steps

[2182] Registration and storage of user information

[2183] Step 1:

[2184] The user opens the Wi-Fi connection settings screen on the tablet device and enters the SSID and password to establish a connection.

[2185] Input: Wi-Fi SSID and password

[2186] Output: Wi-Fi connection establishment status

[2187] Step 2:

[2188] The user enters personal information (name, age, address, etc.) using a tablet device.

[2189] Input: Personal information such as name, age, address, etc.

[2190] Output: Temporarily save the entered personal information

[2191] Step 3:

[2192] The terminal encrypts the information entered and sends it to the server using the SSL / TLS protocol.

[2193] Input: User personal information

[2194] Output: Encrypted transmission of personal information

[2195] Step 4:

[2196] The server decrypts the received information and stores it in a secure database.

[2197] Input: Encrypted personal information

[2198] Output: Personal information stored in the database

[2199] Email Analysis

[2200] Step 1:

[2201] The device periodically checks the user's email account for new emails.

[2202] Input: Email account information

[2203] Output: New emails

[2204] Step 2:

[2205] The device receives new email and retrieves its contents (body, sender, attachments, etc.).

[2206] Input: New email

[2207] Output: Retrieved email content

[2208] Step 3:

[2209] The device encrypts the email content and sends it to the server.

[2210] Input: Email content

[2211] Output: Encrypted email content

[2212] Step 4:

[2213] The server decrypts the received email and analyzes it using machine learning algorithms.

[2214] Input: Encrypted email content

[2215] Output: Analysis results

[2216] Step 5:

[2217] If fraud is suspected, the server generates an alert message and sends it to the terminal.

[2218] Input: Analysis results

[2219] Output: Alert message

[2220] Step 6:

[2221] The device displays an alert to the user.

[2222] Input: Alert message

[2223] Output: User notification

[2224] Audio analysis of call content

[2225] Step 1:

[2226] Users receive suspicious fraudulent calls on their landline or smartphone.

[2227] Input:Incoming call

[2228] Output: Call Start

[2229] Step 2:

[2230] The device records the call in real time.

[2231] Input: Call content

[2232] Output: Recorded audio data

[2233] Step 3:

[2234] The device sends the recorded audio data to the server.

[2235] Input: Recorded audio data

[2236] Output: Transmitted audio data

[2237] Step 4:

[2238] The server converts the received voice data into text.

[2239] Input: Audio data

[2240] Output: The converted text

[2241] Step 5:

[2242] The server uses machine learning algorithms to analyze the text data.

[2243] Input: Translated text

[2244] Output: Analysis results

[2245] Step 6:

[2246] If fraud is suspected, the server generates an alert message and sends it to the terminal.

[2247] Input: Analysis results

[2248] Output: Alert message

[2249] Step 7:

[2250] The device notifies the user of the alert.

[2251] Input: Alert message

[2252] Output: User notification

[2253] Mail analysis

[2254] Step 1:

[2255] The user takes a photo of the mail item using the camera on the tablet device.

[2256] Input: Mail

[2257] Output: Image data

[2258] Step 2:

[2259] The terminal transmits the acquired image data to the server.

[2260] Input: Image data

[2261] Output: Image data sent

[2262] Step 3:

[2263] The server extracts the text from the image.

[2264] Input: Image data

[2265] Output: Extracted text

[2266] Step 4:

[2267] The server analyzes the extracted text using machine learning algorithms.

[2268] Input: Extracted text

[2269] Output: Analysis results

[2270] Step 5:

[2271] If fraud is suspected, the server generates an alert message and sends it to the terminal.

[2272] Input: Analysis results

[2273] Output: Alert message

[2274] Step 6:

[2275] The device notifies the user of the alert.

[2276] Input: Alert message

[2277] Output: User notification

[2278] Sending alerts

[2279] Step 1:

[2280] If fraud is suspected, the server generates an alert message.

[2281] Input: Analysis results

[2282] Output: Alert message

[2283] Step 2:

[2284] The server sends an alert message to the terminal.

[2285] Input: Alert message

[2286] Output: Alert message sent

[2287] Step 3:

[2288] The device will display an alert message on the screen of a smart speaker or tablet, and if necessary, will also notify the smart earphones via voice.

[2289] Input: The alert message sent

[2290] Output: User notification

[2291] This makes it possible to comprehensively detect fraudulent activities carried out through multiple communication means and quickly warn users.

[2292] (Application example 1)

[2293] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2294] In today's world, where users, including the elderly, are at increased risk of falling victim to special frauds, there is a need to prevent fraudulent acts via email, phone calls, and mail. However, it is difficult for users themselves to detect the signs of these frauds, so an efficient fraud prevention system is needed. Furthermore, conventional systems have had issues with the difficulty of real-time monitoring and highly accurate judgment.

[2295] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[2296] In this invention, the server includes means for analyzing received emails and determining whether fraud is suspected, means for acquiring and analyzing phone call content and determining whether fraud is suspected, means for scanning documents with a photographing device and determining whether fraud is suspected, means for sending an alert to a user based on the determination result using a generative AI model, and means for generating a prompt message. This allows users to detect the risk of fraud via email, phone calls, or mail in real time and respond quickly.

[2297] "Means for analyzing received e-mails" refers to a function that analyzes the contents of e-mails received by the system and determines whether there is any suspicion of fraud.

[2298] "Means for acquiring and analyzing call content" refers to a function that acquires the content of a user's call, analyzes that content in real time, and determines whether there is any suspicion of fraud.

[2299] "Means for reading documents with a photographic device" refers to a function that allows a user to photograph a specific document with a photographic device such as a camera, and then determine whether or not there is suspicion of fraud based on the image data.

[2300] A "generative AI model" is an algorithmic model trained using machine learning and deep learning techniques that is used to analyze emails, calls, and documents to identify fraud.

[2301] "Means for generating prompts" refers to a function that uses an AI model to create and present appropriate warning messages or alerts to users based on specific conditions or judgment results.

[2302] "Means for sending alerts" refers to a function that sends an alert or message to a user's device to warn them if fraud is suspected.

[2303] As an embodiment of the present invention, a comprehensive system for protecting users, particularly the elderly, from special frauds is provided. The system includes means for automatically analyzing received emails, phone call contents, mail, etc., determining whether there is any suspected fraud, and sending an alert to the user.

[2304] Registration and storage of user information

[2305] During the initial setup, the user uses a tablet or smartphone to enter information (SSID and password) to connect to their home Wi-Fi. They also enter personal information such as their name, age, and address, and the device sends this information to the server. The server receives the information and registers it in a database.

[2306] Email Analysis

[2307] The email analysis method involves the device monitoring the user's email account and sending received emails to a server. The server then uses a generative AI model to analyze the email content and determine whether it is suspected of being fraudulent. If the analysis results indicate a suspicion of fraud, an alert is sent to the device.

[2308] Audio analysis of call content

[2309] When a user receives a suspected fraudulent call on their landline or smartphone, the device (smart speaker or smart earphone) records the call in real time and sends the audio data to a server. The server converts the audio data into text and uses a generative AI model to determine whether the call is suspected to be fraudulent. If fraud is suspected, the device sends the user an alert saying, "This is a potential fraudulent call. Please hang up immediately."

[2310] Mail analysis

[2311] A user takes a photo of a document, such as mail, with the camera on their tablet or smartphone. The device then sends the captured image data to a server. The server extracts text from the image and analyzes it using a generative AI model to determine whether it is suspected of fraud. If fraud is suspected, the device sends the user an alert saying, "This mail is suspected to be fraudulent. Please be careful."

[2312] Sending alerts

[2313] If the server determines that fraud is suspected, it generates a prompt message and sends it to the device, which then displays the message on the screen of the smart speaker or smartphone and, if necessary, sends a voice notification to the smart earphones.

[2314] Hardware and software used

[2315] Hardware: smartphones, tablets, smart speakers, smart earphones, cameras, microphones

[2316] Software: OCR software (e.g., tesseract-ocr), machine learning models (e.g., scikit-learn), speech recognition software (e.g., Google Speech Recognition API), email processing libraries (e.g., python-email, smtplib), call monitoring software (e.g., Twilio API)

[2317] Prompt Sentence Examples

[2318] Use the following as an example prompt for analyzing suspected fraudulent emails:

[2319] "We have come across a potentially fraudulent email. The contents are as follows:

[2320] From: {sender}

[2321] Subject: {subject}

[2322] Contents: {body}

[2323] Please be careful as this appears to be a scam."

[2324] This will enable users to respond to fraudulent activity in real time, improving safety.

[2325] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[2326] Step 1: Register and save user information

[2327] During the initial setup, a user uses a tablet or smartphone to enter information (SSID and password) to connect to their home Wi-Fi. They also enter personal information such as their name, age, and address, which the device then sends to the server. The server receives the information and registers it in a database. The input here is the user's Wi-Fi information and personal information, and the output is the user information stored in the database.

[2328] Step 2: Capture and parse emails

[2329] The device monitors the user's email account and sends received emails to a server. The server uses a generative AI model to analyze the email content and determine whether it is suspected of fraud. The input here is the content of the received email, and the output is a judgment result on whether it is suspected of fraud.

[2330] Step 3: Audio analysis of the call

[2331] When a user receives a suspected fraudulent call on their landline or smartphone, the device (smart speaker or smart earphone) records the call in real time and sends the audio data to a server. The server converts the audio data into text and uses a generative AI model to determine whether the call is suspected of fraud. The input here is the recorded call content, and the output is the converted text and a determination of whether the call is suspected of fraud.

[2332] Step 4: Image analysis of mail

[2333] A user takes a photo of a document, such as mail, using a tablet or smartphone camera. The device then sends the captured image data to a server. The server extracts text from the image and uses a generative AI model to analyze the text and determine whether it is suspected of fraud. The input here is the image of the mail, and the output is the extracted text and a determination of whether it is suspected of fraud.

[2334] Step 5: Sending an alert

[2335] If the server determines that fraud is suspected, it generates a prompt and an alert message and sends it to the device. The device then displays the warning message on the screen of the smart speaker or smartphone, and if necessary, notifies the smart earphones by voice. The input here is the judgment result regarding the suspected fraud, and the output is the generated prompt and alert message.

[2336] Through these steps, users can detect fraud risks in real time and respond quickly.

[2337] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[2338] As an embodiment for carrying out the present invention, various means of a system (hereinafter referred to as an improved system) that combines a special fraud prevention system with an emotion engine will be described.

[2339] Registration and storage of user information

[2340] During the initial setup, the user uses the tablet device to enter the SSID and password to connect to their home Wi-Fi. After that, they enter personal information such as their name, age, and address, and the device sends this information to the server. The server receives the information and registers it in a database.

[2341] Email analysis and emotion recognition

[2342] The device monitors the user's email account and sends received emails to a server. The server analyzes the email content and determines whether it is suspected of being fraudulent. It also recognizes the user's emotions based on the email content. If the analysis results indicate a suspicion of fraud, it sends an alert to the device.

[2343] Voice analysis and emotion recognition of call content

[2344] When a user receives a suspected fraudulent call on their landline or smartphone, the smart speaker or earphone device records the call in real time and sends the audio data to a server. The server converts the audio data into text and then analyzes the text to determine whether the call is suspected of fraud. The system also recognizes the user's emotions from the call content and sends an alert if there is a change in emotion.

[2345] Mail analysis and emotion recognition

[2346] The user takes a photo of a document, such as mail, using the tablet's camera. The device then sends the captured image data to a server. The server extracts text from the image and analyzes it to determine whether it is suspected of fraud. In addition, the system recognizes the user's emotions based on the document's content. If it determines that the document is suspected of fraud, the device sends an alert to the user.

[2347] Sending alerts

[2348] If fraud is suspected, the server generates an alert message and sends it to the device. The device then displays the message on the screen of the smart speaker or tablet, and if necessary, sends a voice notification to the smart earphones. The emotion engine also notifies the user of the analysis results and provides appropriate advice based on the user's psychological state.

[2349] Specific examples

[2350] Specific examples of mail analysis

[2351] 1. The user takes a photo of mail delivered to their home with their camera.

[2352] 2. The device sends the image data to the server.

[2353] 3. The server extracts text from the image and analyzes the text using machine learning algorithms.

[2354] 4. The server further recognizes the user's emotions based on the document content.

[2355] 5. If it determines that there is a suspicion of fraud, the device will send the user an alert saying, "This mail is suspected to be fraudulent. Please be careful," and will also send an emotional alert saying, "This mail conveys a sense of anxiety. Please ignore it."

[2356] Specific examples of call analysis

[2357] 1. A user receives a suspected fraudulent call on their landline.

[2358] 2. The smart speaker device records the call and sends the data to the server.

[2359] 3. The server converts the speech into text and analyzes it.

[2360] 4. The server recognizes the user's emotions from the content of the call and sends an alert if there is a change in emotions.

[2361] 5. If fraud is suspected, the device will send the user an alert saying, "This is a potential fraudulent call. Please hang up immediately," and will also provide an emotional alert saying, "You are feeling surprised and anxious. Please hang up."

[2362] As a result, the improved system can prevent the threat of special fraud and ensure safety for users, including the elderly. In addition, it can recognize the user's emotional state in real time and provide psychological support.

[2363] The processing flow will be explained below.

[2364] Email analysis and emotion recognition

[2365] Step 1:

[2366] The user receives an email.

[2367] Step 2:

[2368] The terminal acquires the contents of the received email and sends them to the server.

[2369] Step 3:

[2370] The server analyzes the content of the email to determine whether it is suspected of being fraudulent.

[2371] Step 4:

[2372] The server recognizes the user's emotions based on the email content.

[2373] Step 5:

[2374] The server sends the analysis results and emotion recognition results to the device.

[2375] Step 6:

[2376] If fraud is suspected, the device will display or notify the user of an alert and an emotion alert.

[2377] Voice analysis and emotion recognition of call content

[2378] Step 1:

[2379] A user initiates a call on a desk phone or smartphone.

[2380] Step 2:

[2381] The device, a smart speaker or smart earphones, records the call in real time.

[2382] Step 3:

[2383] The device sends the recorded audio data to the server.

[2384] Step 4:

[2385] The server converts the audio data into text.

[2386] Step 5:

[2387] The server analyzes the converted text to determine whether it is suspected of fraud.

[2388] Step 6:

[2389] The server recognizes the user's emotions based on the contents of the call.

[2390] Step 7:

[2391] The server sends the analysis results and emotion recognition results to the device.

[2392] Step 8:

[2393] If fraud is suspected, the device will display an alert and emotion alert to the user or notify them audibly during the call.

[2394] Mail analysis and emotion recognition

[2395] Step 1:

[2396] The user takes a photo of the mail that arrives at their home using the camera on their tablet device.

[2397] Step 2:

[2398] The terminal transmits the captured image data to the server.

[2399] Step 3:

[2400] The server extracts the text from the image.

[2401] Step 4:

[2402] The server analyzes the extracted text to determine whether it is suspected of fraud.

[2403] Step 5:

[2404] The server recognizes the user's emotions based on the document content.

[2405] Step 6:

[2406] The server sends the analysis results and emotion recognition results to the device.

[2407] Step 7:

[2408] If fraud is suspected, the device will display or notify the user of an alert and an emotion alert.

[2409] account settings

[2410] Step 1:

[2411] The user opens the initial setup screen using the tablet device.

[2412] Step 2:

[2413] The user enters the Wi-Fi SSID and password.

[2414] Step 3:

[2415] The device attempts to connect to Wi-Fi and displays the results to the user.

[2416] Step 4:

[2417] The user enters personal information such as name, age, and address.

[2418] Step 5:

[2419] The terminal transmits the input information to the server.

[2420] Step 6:

[2421] The server receives the information and stores it in a database.

[2422] Step 7:

[2423] The server sends a confirmation message to the terminal that the save has been completed, and the terminal notifies the user.

[2424] Through these processing steps, the improved system can ensure the safety of users and prevent them from committing fraud. Furthermore, by utilizing the emotion engine, it is possible to determine the user's emotional state in real time and provide psychological support.

[2425] Example 2

[2426] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2427] In modern society, special frauds are becoming increasingly sophisticated and their methods are diversifying. Fraud targeting the elderly in particular has become a major social problem, and conventional fraud prevention systems are often unable to keep up with new methods. Furthermore, the lack of appropriate countermeasures and support that take into account the user's emotional state makes it difficult to prevent damage before it occurs. To solve this problem, it is necessary to provide a system that can effectively analyze communication methods (email, phone calls, documents) that are suspected of being fraudulent, and that can recognize and respond to the user's emotional state in real time.

[2428] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[2429] In this invention, the server includes means for analyzing received electronic messages and determining whether fraud is suspected, means for acquiring and analyzing the contents of phone calls and determining whether fraud is suspected, means for photographing and reading documents and determining whether fraud is suspected, means for analyzing the emotional state of the user, and means for sending an alert to the user based on the determination result and the emotional state. This makes it possible to deal with a variety of fraudulent methods, provide appropriate support that takes into account the psychological state of the user, and prevent fraud damage before it occurs.

[2430] "Means for analyzing received electronic messages to determine whether they are suspected of fraud" refers to a combination of software and hardware that processes the content of emails and messages received by a user and automatically detects possible fraudulent activity.

[2431] "Means for acquiring and analyzing call content to determine whether fraud is suspected" refers to a combination of software and hardware that records the user's call content and analyzes the audio data to identify possible fraud.

[2432] "Means for photographing and reading documents to determine whether they are suspected of fraud" refers to a combination of software and hardware that converts the contents of a document photographed by a user with a camera into text data using technologies such as optical character recognition, and then analyzes the text data to determine whether it is suspected of fraud.

[2433] The "means for analyzing a user's emotional state" is a combination of software and hardware for analyzing a user's emotional state based on the contents of electronic messages, conversations and documents, and evaluating the results.

[2434] The "means for sending an alert to the user based on the judgment result and emotional state" refers to a combination of software and hardware that, when a suspected fraud is judged to be present, generates a warning message that takes into account the judgment result and the user's emotional state and notifies the user.

[2435] "Means for registering and storing user information" refers to a combination of software and hardware for registering users' personal information in a database and using or referencing it when necessary.

[2436] "Means for monitoring a user's voice communications in real time and sending an alert if the communications contain suspected fraudulent content" refers to a combination of software and hardware that monitors a user's calls in real time and immediately issues an alert if suspected fraudulent content is detected.

[2437] As an embodiment of the present invention, the detailed configuration and operation of a system (hereinafter referred to as "this system") that combines a special fraud prevention system with an emotion recognition engine will be described.

[2438] Registration and storage of user information

[2439] The user uses the tablet device for initial setup and enters the SSID and password to connect to their home Wi-Fi. Next, they enter personal information such as their name, age, and address into the device. This information is sent from the device to the server. The server registers the received information in a database, which allows the user's basic information to be safely stored.

[2440] Analysis of electronic messages and emotion recognition

[2441] The device periodically monitors the user's email account and receives new emails. The received emails are immediately sent to the server. The server uses a text analysis engine to analyze the email content and determine whether it is suspected of being fraudulent. The server also uses an emotion recognition engine to extract the user's emotions from the email content. If it determines that the email is suspected of being fraudulent, it sends an alert message to the device and displays a warning to the user.

[2442] Voice analysis and emotion recognition of call content

[2443] When a user receives a suspected fraudulent call on their landline or smartphone, the smart speaker or earphones that are part of the device record the call in real time. The recorded data is then immediately sent to a server. The server uses a speech recognition engine to convert the voice data into text and analyzes the text to determine whether the call is suspected of fraud. The server also recognizes the user's emotions from the call content and sends an alert if a change in emotion is detected.

[2444] Document Analysis and Emotion Recognition

[2445] The user takes a photo of a document, such as mail, with the tablet's camera. The device then sends the captured image data to the server. The server uses an image analysis engine to extract text from the image and analyzes that text to determine whether it is suspected of fraud. The server then recognizes the user's emotions based on the document's content. If it determines that the document is suspected of fraud, the device sends an alert to the user.

[2446] Sending alerts

[2447] If a suspected fraud attempt is detected, the server generates an alert message and sends it to the device, which then displays the message on the screen of the smart speaker or tablet and, if necessary, also notifies the smart earphones via voice notification, allowing the user to immediately become aware of the possibility of fraud.

[2448] Specific examples

[2449] Specific examples of mail analysis

[2450] 1. The user takes a photo of the mail that arrives at their home with their camera.

[2451] 2. The device sends the image data to the server.

[2452] 3. The server extracts text from the image and analyzes the text using machine learning algorithms.

[2453] 4. The server further recognizes the user's emotions based on the document content.

[2454] 5. If it determines that there is a suspicion of fraud, the device will send the user an alert saying, "This mail is suspected to be fraudulent. Please be careful," and will also send an emotional alert saying, "This mail conveys a sense of anxiety. Please ignore it."

[2455] Specific examples of call analysis

[2456] 1. A user receives a suspected fraudulent call on their landline.

[2457] 2. The smart speaker device records the call and sends the data to the server.

[2458] 3. The server converts the speech into text and analyzes it.

[2459] 4. The server recognizes the user's emotions from the content of the call and sends an alert if there is a change in emotion.

[2460] 5. If fraud is suspected, the device will send the user an alert saying, "This is a potential fraudulent call. Please hang up immediately," and will also provide an emotional alert saying, "You are feeling surprised and anxious. Please hang up."

[2461] In this way, this system can prevent the threat of special fraud and ensure the safety of users, including the elderly. In addition, it can recognize the user's emotional state in real time and provide psychological support.

[2462] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2463] Registration and storage of user information

[2464] Processing Steps

[2465] Step 1:

[2466] The user opens the settings screen on the tablet device and enters the SSID and password to connect to their home Wi-Fi.

[2467] Input: SSID, password

[2468] Output: Wi-Fi connection complete message

[2469] Specific operation: The user enters the SSID and password in the specified fields and presses the "Connect" button. The device receives this and attempts to connect to Wi-Fi. If the connection is successful, a connection completion message is displayed.

[2470] Step 2:

[2471] The user enters personal information such as name, age, and address into the terminal.

[2472] Input: Personal information such as name, age, address, etc.

[2473] Output: Input confirmation screen

[2474] Specific operation: The user enters personal information into each field and presses the "Submit" button. The device retrieves this information and displays a confirmation screen.

[2475] Step 3:

[2476] The terminal transmits the entered personal information to the server.

[2477] Input: Personal information entered on the device

[2478] Output: Message that data has been sent to the server

[2479] Specific operation: The terminal packetizes the entered personal information and sends it to the server using a secure communication protocol (e.g., HTTPS).

[2480] Step 4:

[2481] The server registers the received personal information in a database.

[2482] Input: Personal information sent from the device

[2483] Output: Database registration completion message

[2484] Specific operation: The server analyzes the received data and stores it in the database appropriately. Once the data has been stored, it returns a registration completion message to the device.

[2485] Analysis of electronic messages and emotion recognition

[2486] Processing Steps

[2487] Step 1:

[2488] The device periodically monitors the user's email account and receives new emails.

[2489] Input: Email login details

[2490] Output: New email information

[2491] Specific operation: The device sends a query to the mail server at the specified interval to check for new mail. If new mail is found, the device retrieves its contents.

[2492] Step 2:

[2493] The device sends the contents of the new email to the server.

[2494] Input: New email content

[2495] Output: Message that email has been sent to the server

[2496] Specific operation: The device obtains the body and header information of the new email and sends it to the server.

[2497] Step 3:

[2498] The server uses a text analysis engine to analyze the email content and determine whether it is suspected of being fraudulent.

[2499] Input: Email text data

[2500] Output: Fraud detection result

[2501] What it does: The server uses a text analysis engine (e.g., an NLP model) to analyze the content of the email and determine whether it is fraudulent. The result is stored as a flag.

[2502] Step 4:

[2503] The server uses an emotion recognition engine to extract the user's emotion from the content of the email.

[2504] Input: Email text data

[2505] Output: Emotion recognition result

[2506] Specific operation: The server uses an emotion recognition engine (e.g., emotion analysis API) to extract the user's emotion from the email body and saves the result as a flag.

[2507] Step 5:

[2508] If fraud is suspected, an alert message will be sent to the terminal and a warning will be displayed to the user.

[2509] Input: Fraud detection result, emotion recognition result

[2510] Output: Alert message screen

[2511] Specific operation: If the fraud flag is set, the server generates an alert message and sends it to the terminal. The terminal receives it and displays the warning message to the user.

[2512] Voice analysis and emotion recognition of call content

[2513] Processing Steps

[2514] Step 1:

[2515] Users receive suspicious fraudulent calls on their landline or smartphone.

[2516] Input: Incoming call notification

[2517] Output: Call start signal

[2518] Specific operation: The user receives an incoming call and answers the phone. The device detects this and transitions to a call state.

[2519] Step 2:

[2520] Smart speakers and smart earphones that are part of the device record the contents of calls in real time.

[2521] Input: Call audio data

[2522] Output: Recording data

[2523] Specific operation: The terminal starts recording as soon as the call starts and stores the audio data in a buffer.

[2524] Step 3:

[2525] The recording data is immediately sent to the server.

[2526] Input: Recording data

[2527] Output: Message that data has been sent to the server

[2528] What happens: The device sends the recording data to the server using a secure protocol (e.g., HTTPS).

[2529] Step 4:

[2530] The server uses a speech recognition engine to convert the voice data into text.

[2531] Input: Audio data

[2532] Output: Text data

[2533] Specific operation: The server uses a speech recognition engine (e.g., Speech-to-Text API) to convert the voice data into text.

[2534] Step 5:

[2535] The server analyzes the text to determine if it is suspected of fraud.

[2536] Input: Converted text data

[2537] Output: Fraud detection result

[2538] What it does: The server uses a text analysis engine to determine the likelihood of fraud, and stores the result as a flag.

[2539] Step 6:

[2540] The server recognizes the user's emotions from the content of the call and sends an alert if there is a change in emotion.

[2541] Input: Text data

[2542] Output: Emotion recognition result

[2543] Specific operation: The server uses an emotion recognition engine to extract the user's emotion from the text data and saves the result as a flag.

[2544] Step 7:

[2545] If fraud is suspected, an alert message will be sent to the terminal and a warning will be displayed to the user.

[2546] Input: Fraud detection result, emotion recognition result

[2547] Output: Alert message screen

[2548] Specific operation: If the fraud flag is set, the server generates an alert message and sends it to the terminal. The terminal receives it and displays the warning message to the user.

[2549] Document Analysis and Emotion Recognition

[2550] Processing Steps

[2551] Step 1:

[2552] The user takes a photo of the mail item using the camera on the tablet device.

[2553] Input: Image data of mail

[2554] Output: Shooting completion message

[2555] Specific operation: The user takes a photo of the mail using the camera function of the tablet device. The image data is stored on the device.

[2556] Step 2:

[2557] The terminal transmits the image data to the server.

[2558] Input: Image data of mail

[2559] Output: Message that data has been sent to the server

[2560] Specific operation: The device sends the acquired image data to the server using a secure protocol (e.g., HTTPS).

[2561] Step 3:

[2562] The server uses an image analysis engine to extract text from the image.

[2563] Input: Image data

[2564] Output: Extracted text data

[2565] Specific operation: The server uses an image analysis engine (e.g., OCR API) to extract text data from the image.

[2566] Step 4:

[2567] The server analyzes the text to determine if it is suspected of fraud.

[2568] Input: Extracted text data

[2569] Output: Fraud detection result

[2570] What it does: The server uses a text analysis engine to determine the likelihood of fraud, and stores the result as a flag.

[2571] Step 5:

[2572] The server recognizes the user's emotions based on the document content.

[2573] Input: Extracted text data

[2574] Output: Emotion recognition result

[2575] Specific operation: The server uses an emotion recognition engine to extract the user's emotion from the document content, and the result is saved as a flag.

[2576] Step 6:

[2577] If fraud is suspected, an alert message will be sent to the terminal and a warning will be displayed to the user.

[2578] Input: Fraud detection result, emotion recognition result

[2579] Output: Alert message screen

[2580] Specific operation: If the fraud flag is set, the server generates an alert message and sends it to the terminal. The terminal receives it and displays a warning message to the user.

[2581] Sending alerts

[2582] Processing Steps

[2583] Step 1:

[2584] The server detects suspected fraud.

[2585] Input: Fraud detection result, emotion recognition result

[2586] Output: Alert message

[2587] Specific operation: The server checks the fraud determination flag and emotion recognition flag and generates an alert message if necessary.

[2588] Step 2:

[2589] The server generates an alert message and sends it to the terminal.

[2590] Input: Fraud detection result, emotion recognition result

[2591] Output: The generated alert message

[2592] Specific operation: The server generates an alert message and sends it to the terminal in an appropriate format.

[2593] Step 3:

[2594] The device displays or notifies you of an alert message.

[2595] Input: The generated alert message

[2596] Output: The displayed warning message screen

[2597] Specific operation: When the terminal receives an alert message, it displays a visual or audio warning message to the user.

[2598] Through the above processing steps, the system can provide high detection accuracy and rapid response to a variety of fraudulent methods, while also providing appropriate support based on the user's emotional state.

[2599] (Application example 2)

[2600] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2601] In recent years, there has been an increase in special frauds targeting the elderly, and many of these are committed via email, telephone, or mail. This has led to a growing need for systems that can not only detect fraud but also monitor users' emotions in real time and provide appropriate responses based on their psychological state. While conventional systems specialize in detecting fraud, they do not provide support that takes users' emotions into consideration, resulting in a lack of psychological support.

[2602] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2603] In this invention, the server includes means for analyzing received electronic documents and determining whether fraud is suspected, means for acquiring and analyzing the contents of phone calls and determining whether fraud is suspected, means for scanning documents with an imaging device and determining whether fraud is suspected, means for recognizing the user's emotions using an emotion recognition engine when determining whether fraud is suspected, and providing appropriate advice on how to respond if there is a possibility of fraud, and means for sending a warning to the user based on the determination result. This makes it possible not only to respond immediately to information that is suspected of being fraud, but also to provide psychological support that takes into account the user's emotional state.

[2604] An "electronic document" is a document stored in digital format, including email, PDF, Word files, etc.

[2605] "Call content" refers to information exchanged through voice communication, such as the content of a conversation conducted via telephone or internet telephone.

[2606] "Photographing device" refers to equipment for taking still or moving images, and includes devices with camera functions.

[2607] An "emotion recognition engine" is a combination of software and hardware for identifying and analyzing human emotional states.

[2608] A "warning" is a message or notification that warns the user of a caution or danger based on a specific condition.

[2609] "User Information" refers to personal data relating to users of the system, including name, age, address, contact details, etc.

[2610] "Analysis" refers to the process of examining a set of data or information in detail to understand its content, trends, and relationships.

[2611] "Judgment" is the act of determining an outcome based on collected data and information according to specific conditions and criteria.

[2612] System Overview

[2613] This system is a security system that combines fraud prevention and user emotion recognition. The system analyzes the contents of received electronic documents, phone calls, and documents captured using a camera, and issues a warning if fraud is suspected. Furthermore, the system uses an emotion recognition engine to analyze the user's emotional state and provide appropriate responses.

[2614] Hardware and software used

[2615] Hardware:

[2616] Smartphone

[2617] camera

[2618] software:

[2619] Server side: Django

[2620] Emotion Recognition API:Azure Cognitive Services Emotion API

[2621] NLP libraries: spaCy, nltk

[2622] Deep Learning: TensorFlow, PyTorch

[2623] Speech Recognition: Google Speech-to-Text API

[2624] OCR library: Tesseract

[2625] Program processing

[2626] 1. Electronic Document Analysis:

[2627] The server periodically retrieves emails received via the API and performs text analysis using an NLP library (spaCy or nltk).

[2628] It analyzes text structure and keywords, and if there is a possibility of fraud, it uses the Azure Cognitive Services Emotion API to analyze the user's emotions.

[2629] If fraud is suspected, a warning message will be generated and sent to the user's smartphone.

[2630] 2. Call analysis:

[2631] It monitors smartphone calls in real time and converts audio data into text using the Google Speech-to-Text API.

[2632] Use NLP libraries to analyze text and determine potential fraud.

[2633] The Azure Cognitive Services Emotion API analyzes the user's emotions, and if anxiety or surprise is detected, a warning message and emotion-based advice is sent to the user.

[2634] 3. Document Analysis:

[2635] A photograph of a document such as mail is taken using a photographing device (smartphone camera), and the image data is sent to the server.

[2636] The server uses an OCR library (Tesseract) to extract text from image data and performs text analysis using an NLP library.

[2637] If fraud is suspected, emotion recognition is performed, a message advising appropriate action is generated, and a warning is sent to the user's smartphone.

[2638] Specific examples

[2639] Specific examples of email analysis

[2640] Process flow: Retrieve email from SMTP server → Analyze text using spaCy → Analyze emotions using Azure Cognitive Services Emotion API → Warn user if fraud is suspected

[2641] Example alerts: "This email may be fraudulent. Please be cautious.", "This email makes me feel uneasy. Please ignore it."

[2642] Prompt Sentence Examples

[2643] "Please analyze the content of the email below to determine the likelihood of a scam and the associated emotions.

[2644] The email reads: "Dear customer, you have been selected for our special campaign. Please update your account information to receive your prize."

[2645] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2646] Step 1: Obtaining electronic documents

[2647] Description: The server periodically retrieves the user's email via API.

[2648] Input: User's email account information

[2649] Output: Retrieved email data

[2650] What happens: The server fetches emails through the SMTP server's API and retrieves new emails in the inbox.

[2651] Step 2: Text analysis

[2652] Description: The server performs text analysis on the body of the retrieved email.

[2653] Input: Email data

[2654] Output: Analysis results (suspected fraud)

[2655] What it does: The server uses an NLP library (e.g. spaCy) to tokenize the email body, extract keywords, and perform contextual analysis to assess the likelihood of fraud.

[2656] Step 3: Sentiment Analysis

[2657] Description: The server analyzes the user's emotional state based on the content of the email.

[2658] Input: Email data, NLP analysis results

[2659] Output: Emotion analysis results

[2660] How it works: The server uses the Azure Cognitive Services Emotion API to estimate the emotions the user may be feeling from the content of the email.

[2661] Step 4: Generate warning messages

[2662] Description: If fraud is suspected, the server generates a warning message.

[2663] Input: NLP analysis results, sentiment analysis results

[2664] Output: Warning message

[2665] Specific behavior: The server generates a warning message based on the analysis results, for example, "This email may be fraudulent. Please be careful."

[2666] Step 5: Sending an alert

[2667] Description: The server sends the generated warning message to the user's terminal.

[2668] Input: warning message

[2669] Output: A warning displayed on the user's terminal

[2670] Specific operation: The server sends a warning message to the user's smartphone via push notification.

[2671] Step 6: Capture the call

[2672] Description: The device receives the user's call content in real time.

[2673] Input: Call audio data

[2674] Output: Recorded audio data

[2675] How it works: The smartphone uses its built-in microphone to record the conversation and sends it to the server as audio data.

[2676] Step 7: Speech to Text

[2677] Description: The server converts the call audio data into text.

[2678] Input: Captured audio data

[2679] Output: Text data

[2680] Specific operation: The server uses the Google Speech-to-Text API to convert the captured voice data into text.

[2681] Step 8: Call Analysis

[2682] Description: The server analyzes the text of the call to determine if it is suspected of fraud.

[2683] Input: Text data

[2684] Output: Analysis results (suspected fraud)

[2685] What it does: The server uses an NLP library to tokenize the call, extract keywords and perform contextual analysis to assess the likelihood of fraud.

[2686] Step 9: Sentiment Analysis

[2687] Description: The server analyzes the user's emotional state from the content of the call.

[2688] Input: Text data, NLP analysis results

[2689] Output: Emotion analysis results

[2690] Specific operation: The server uses the Azure Cognitive Services Emotion API to estimate the user's emotions from the content of the call.

[2691] Step 10: Generate warning messages

[2692] Description: If fraud is suspected or there is a change in sentiment, the server generates a warning message.

[2693] Input: NLP analysis results, sentiment analysis results

[2694] Output: Warning message

[2695] Specific behavior: Based on the analysis results, the server generates a warning message such as "This call may be fraudulent. Please hang up immediately."

[2696] Step 11: Sending an alert

[2697] Description: The server sends the generated warning message to the user's terminal.

[2698] Input: warning message

[2699] Output: A warning displayed on the user's terminal

[2700] Specific operation: The server sends a warning message to the user's smartphone via push notification.

[2701] Step 12: Photograph the document

[2702] Description: A user takes a picture of a document, such as mail, with a camera.

[2703] Input: Image data

[2704] Output: Photographed document image

[2705] Specific action: The user uses the smartphone camera to take a picture of the mail.

[2706] Step 13: Extracting text from images

[2707] Description: The server extracts text from the captured image.

[2708] Input: Image data

[2709] Output: Extracted text data

[2710] Specific operation: The server uses an OCR library (Tesseract) to extract text from the image.

[2711] Step 14: Document Content Analysis

[2712] Description: The server analyzes the text data to determine if it is suspected of fraud.

[2713] Input: Text data

[2714] Output: Analysis results (suspected fraud)

[2715] How it works: The server uses an NLP library to tokenize the document content, extract keywords and perform contextual analysis to assess the likelihood of fraud.

[2716] Step 15: Sentiment Analysis

[2717] Description: The server analyzes the user's emotional state from the document content.

[2718] Input: Text data, NLP analysis results

[2719] Output: Emotion analysis results

[2720] Specific operation: The server uses the Azure Cognitive Services Emotion API to estimate the user's emotions from the document content.

[2721] Step 16: Generate warning messages

[2722] Description: If fraud is suspected, the server generates a warning message.

[2723] Input: NLP analysis results, sentiment analysis results

[2724] Output: Warning message

[2725] Specific operation: Based on the analysis results, the server creates a message such as "This mail is suspected to be fraudulent. Please be careful."

[2726] Step 17: Sending Alerts

[2727] Description: The server sends the generated warning message to the user's terminal.

[2728] Input: warning message

[2729] Output: A warning displayed on the user's terminal

[2730] Specific operation: The server sends a warning message to the user's smartphone via push notification.

[2731] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2732] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2733] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2734] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2735] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circ...

Claims

1. a means for analyzing received email to determine whether it is suspected of being fraudulent; A means for capturing and analyzing the content of calls to determine whether fraud is suspected; and A means of scanning documents with a camera to determine whether they are suspected of fraud; means for sending an alert to a user based on the determination result; A system including:

2. 10. The system of claim 1, further comprising means for registering and storing user information.

3. 10. The system of claim 1, further comprising means for monitoring a user's calls in real time and sending an alert if the calls contain suspected fraudulent content.

Images