System

Abstract

A system is provided.SOLUTION: A system comprising: means for recording call content; means for analyzing the recorded call content with a generated AI model; means for determining a likelihood of special fraud based on an analysis result; and means for sending a notification if there is a likelihood of special fraud.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The methods of special fraud are becoming more diverse, and fraud targeting the elderly in particular is on the rise. This presents a challenge for the elderly, as they have difficulty immediately recognizing fraud and taking appropriate action. Furthermore, the lack of a reliable network to seek help from when they encounter fraud exacerbates the isolation of the elderly. Therefore, there is a need for a safe and reliable method to prevent fraud and provide an environment where the elderly can live with peace of mind. [Means for solving the problem]

[0005] This invention is a system that records phone conversations, analyzes the recorded conversations using a generative AI model, and determines the possibility of special fraud based on the analysis results. If the determination results indicate the possibility of special fraud, a notification is sent, thereby preventing fraud. Specifically, the system provides a means for scoring the conversations and a means for anonymizing and saving the conversations and recording them as patterns when the possibility of fraud is detected. This system can reduce fraud victimization among the elderly and raise their crime prevention awareness.

[0006] "Call content" refers to all information exchanged in voice communication via a terminal.

[0007] "Recording" refers to the technical means of storing sound on a recording medium.

[0008] A "generative AI model" is a type of artificial intelligence model that uses machine learning techniques to automatically perform specific tasks.

[0009] "Analysis" refers to the process of processing recorded audio data to understand and interpret its content.

[0010] "Special fraud" refers to fraud committed using specific methods, generally targeting elderly people.

[0011] "Likelihood" refers to the probability or degree to which a particular event may occur.

[0012] "Judgment" refers to the process of making decisions based on the results of analysis.

[0013] "Notification" refers to the means or method of communicating specific information to interested parties.

[0014] "Scoring" refers to the method by which a generative AI model quantifies the results of its analysis.

[0015] "Anonymization" refers to a method of processing data so that personal information cannot be identified.

[0016] A "pattern" refers to a characteristic or tendency common to a particular phenomenon.

[0017] "Storage" refers to the act of recording data for later use. [Brief explanation of the drawings]

[0018] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0019] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0020] First, the terms used in the following description will be explained.

[0021] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0022] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0023] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0024] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0025] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0026] [First embodiment]

[0027] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0028] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0029] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0030] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0031] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0032] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0033] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0034] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0035] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0036] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0037] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0038] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0039] This invention is a system that records phone conversations, analyzes the recorded conversations using a generative AI model, determines the possibility of special fraud, and notifies the user. The following describes in detail an embodiment of this system.

[0040] System configuration

[0041] This system consists of the user's smartphone (terminal), a server that stores and analyzes voice data, and a network for sending notifications.

[0042] Terminal

[0043] The terminal has the function of recording voice data during a call and transmitting the data to the server in real time. When a call starts, the voice recording module is automatically started and the voice data is buffered.

[0044] server

[0045] The server receives the voice data sent from the device and processes it in real time. The generative AI model used here analyzes the voice data based on pre-trained fraud characteristics. Based on the analysis results, it scores the likelihood of fraud, and if the score exceeds a pre-set threshold, it is determined that there is a high probability of fraud. The server anonymizes and saves the content of calls that may be fraudulent, recording them as patterns.

[0046] Notification Network

[0047] The notification network sends notifications to users and pre-defined trusted contacts (e.g., family members) when potential fraud is detected via text message, email, or dedicated in-app notifications.

[0048] Program processing

[0049] Below we explain how each component of the system works.

[0050] Start recording a call (device)

[0051] When a user initiates a call on their smartphone, the device automatically records the call, storing the recording in a temporary buffer and sending it to a server in real time.

[0052] Receiving and analyzing voice data (server)

[0053] The server receives the voice data sent from the device and begins analysis using the generative AI model. The generative AI model converts the recorded data into text using a speech recognition module, and analyzes the text data to determine whether it contains any signs of fraud. As a result of the analysis, a score is assigned to the likelihood of fraud, and if the score is high, it is determined to be fraud.

[0054] Possible fraud notification (server)

[0055] If a call is deemed likely to be fraudulent, the server will send a notification to a pre-defined contact (either the user or one of their trusted contacts) stating that a potentially fraudulent call has been detected.

[0056] Save and patternize call content (server)

[0057] If a call is deemed potentially fraudulent, it is anonymized and stored in a pattern database that records the characteristics of fraud. This data will then be used to develop fraud prevention education materials and improve future AI models. Users and their families will receive regular updates on the latest fraud scams and prevention strategies.

[0058] Specific examples

[0059] System Operation Scenarios

[0060] For example, if user A receives a call from a self-proclaimed "bank representative" while on a call on his smartphone, the system operates as follows.

[0061] 1. When user A answers the phone, the device automatically records the call and sends the recording data to the server in real time.

[0062] 2. The server receives the audio data and the generative AI model analyzes it to ensure that the call contains typical scam keywords such as "bank," "account number," and "transfer."

[0063] 3. The generative AI model produces a high fraud score (e.g., 93%). The server determines that fraud is likely and prepares to send a notification.

[0064] 4. The server notifies User A and his / her family via text message that a potentially fraudulent call has been detected.

[0065] 5. The call content is anonymized and stored in a fraud pattern database.

[0066] 6. User A and his family will receive a report at a later date containing updated information on fraud prevention.

[0067] This system will quickly protect elderly people from fraud and enable them to take appropriate crime prevention measures, thereby providing an environment in which they can live their daily lives with peace of mind.

[0068] The processing flow will be explained below.

[0069] Step 1: Start recording the call (on your device)

[0070] When a user starts a call on the smartphone, the terminal detects a call start event.

[0071] The device will automatically start the audio recording module.

[0072] The device stores the recorded data in a temporary buffer and prepares it for transmission to the server in real time.

[0073] Step 2: Receiving audio data (server)

[0074] The server receives the voice data packets sent from the terminal.

[0075] The received audio data is stored in temporary storage.

[0076] The server queues the audio data for analysis by the generative AI model.

[0077] Step 3: Analyzing the voice data (server)

[0078] The server retrieves the audio data from the analysis queue.

[0079] The server uses voice recognition technology to convert the voice data into text data.

[0080] The server inputs the converted text data into the generative AI model.

[0081] The generative AI model analyzes the fraud features and calculates a fraud score.

[0082] Step 4: Determine the possibility of fraud (server)

[0083] The server evaluates the fraud scores returned by the generative AI model.

[0084] Determine whether the score exceeds a set threshold.

[0085] If fraud is deemed likely, prepare to notify.

[0086] Step 5: Sending Notifications (Server)

[0087] The server generates a notification message if fraud is deemed likely.

[0088] Notifications will be sent to the user and pre-registered contacts (family members, etc.).

[0089] Notifications will be sent via text message and in-app notifications.

[0090] Step 6: Sharing call content and patterning (server)

[0091] The server performs a process to anonymize the content of calls that are determined to be highly likely to be fraudulent.

[0092] Anonymized call transcripts are stored in a fraud pattern database.

[0093] The latest fraud methods are analyzed based on the collected data.

[0094] Regularly provide fraud prevention educational information to you and your family.

[0095] Example 1

[0096] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0097] In recent years, the methods used by special fraudsters targeting the elderly have become more sophisticated, resulting in an increase in the number of victims. To deal with such situations, it is necessary to analyze call content in real time and quickly identify dangerous calls. However, with conventional methods, recording and analyzing call content is time-consuming and laborious, making it difficult to respond quickly. In addition, there is a lack of systems that can effectively identify possible fraud and notify users in a reliable manner. To solve these issues, it is necessary to provide a system that can perform an integrated process from recording call content to detecting fraud, notifying users, and storing data.

[0098] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0099] In this invention, the server includes means for automatically recording the contents of a call when the user initiates a call, means for transmitting the recorded voice data to the server in real time, means for converting the received voice data into text data using voice recognition technology, means for analyzing the converted text data using a generative AI model to determine the possibility of fraud, means for sending a notification to the user and a designated third party if there is a possibility of fraud, and means for anonymizing and saving the contents of a call that is likely to be fraudulent and recording it in a fraud pattern database, thereby enabling real-time analysis of the contents of the call and rapid detection of fraud.

[0100] The "means for automatically recording telephone conversations" refers to a device or software that has the function of automatically recording telephone conversations when a user initiates a call and temporarily storing the recorded conversations in a buffer.

[0101] "Means for transmitting recorded voice data to a server in real time" refers to a device or software that has the function of transmitting recorded voice data to a server in real time at regular intervals or for each data size.

[0102] "Means for converting received voice data into text data using voice recognition technology" refers to a device or software that includes a voice recognition module or API for analyzing received voice data and converting its contents into text.

[0103] A "means for analyzing text data converted using a generative AI model to determine the likelihood of fraud" is a device or software that includes algorithms or methods for analyzing text data using generative AI to identify characteristics of fraud.

[0104] "Means for sending notifications to the user and designated third parties in the event of possible fraud" refers to a device or software that has the functionality to send notifications to pre-defined contacts in the event that it is determined that there is a high possibility of fraud.

[0105] "Means for anonymizing and storing phone call content that is likely to be fraudulent and recording it in a fraud pattern database" refers to a device or software that has the function of anonymizing phone call content that is determined to be likely to be fraudulent and storing the characteristics of such phone call in a database.

[0106] This invention is a system that records phone conversations, analyzes the recorded conversations using a generative AI model, determines the possibility of special fraud, and notifies the user. Specific embodiments for implementing this system are described in detail below.

[0107] System configuration

[0108] This system consists of a user's communication device (e.g., a smartphone), a server that stores and analyzes voice data, and a communication network for sending notifications.

[0109] communication terminal

[0110] When a user starts a call, the communication device automatically records the call. The recorded voice data is sent to the server in real time. The device is equipped with a voice recognition module, which automatically buffers the voice data when a call starts.

[0111] server

[0112] The server receives and processes the voice data sent from the device. The server is equipped with speech recognition technology using the Google Cloud Speech-to-Text API, which converts the voice data into text data. The converted text data is analyzed by OpenAI's generative AI model (e.g., the GPT series). This generative AI model detects keywords and phrases characteristic of fraud and scores the likelihood of fraud. If the score exceeds a set threshold, the call is deemed to have a high probability of fraud. The content of calls determined to be fraudulent is anonymized and stored in a fraud pattern database.

[0113] Notification Network

[0114] The notification network will notify users and pre-defined trusted contacts (e.g., family members) about potential fraud via text message, email, or dedicated in-app notifications.

[0115] Specific examples

[0116] System Operation Scenarios

[0117] For example, if user A receives a call from someone claiming to be a "bank representative" while on a call on his smartphone, the system operates as follows:

[0118] 1. When user A answers the phone, the device automatically records the call and sends the recording data to the server in real time.

[0119] 2. The server receives the audio data and converts it into text using the Google Cloud Speech-to-Text API.

[0120] 3. The text data is analyzed using OpenAI's generative AI model (e.g., the GPT series). For example, it checks whether the call content contains keywords such as "bank," "account number," or "transfer."

[0121] 4. The generative AI model generates a high fraud score based on the analysis results. For example, if the fraud score is evaluated as 93%, the server determines that there is a high probability of fraud and prepares to send a notification.

[0122] 5. The server notifies User A and his / her family via text message that a potentially fraudulent call has been detected.

[0123] 6. The call content is anonymized and stored in a fraud pattern database.

[0124] 7. User A and his family will receive a report at a later date containing updated information on fraud prevention.

[0125] Example prompt for a generative AI model:

[0126] Analyze calls and score them for potential fraud.

[0127] Caller details: "Hello, I'm a bank representative. What's your account number?"

[0128] This system will quickly protect elderly people from fraud and enable them to take appropriate crime prevention measures, thereby providing an environment in which they can live their daily lives with peace of mind.

[0129] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0130] Step 1: Start recording the call (on your device)

[0131] When a user starts a call on their smartphone, the device's recording module automatically starts and records the call. The recording data is temporarily stored in a buffer.

[0132] Input: Call start signal

[0133] Data processing: The recording module buffers the call audio

[0134] Output: Recorded audio data

[0135] Specific behavior: When the user presses the call button in the phone app, the recording module starts in the background and begins recording the call audio.

[0136] Step 2: Sending audio data (terminal)

[0137] The recorded audio data is sent to the server in real time using a secure encryption protocol (e.g., TLS).

[0138] Input: Buffered audio data

[0139] Data processing: Transmitted to the server in real time (encrypted)

[0140] Output: Audio data sent to the server

[0141] Specific operation: Every time the recording data reaches a certain size (e.g., 5 seconds), the device sends this data to the server.

[0142] Step 3: Receiving audio data (server)

[0143] The server receives the voice data sent from the terminal and adds it to the processing queue. The received data is encrypted to ensure security.

[0144] Input: Audio data sent from the device

[0145] Data processing: Decode received data and add it to the processing queue

[0146] Output: Audio data added to the processing queue

[0147] Specific operation: The server analyzes the arriving data packet, decodes it, and then queues it for processing.

[0148] Step 4: Converting audio data to text (server)

[0149] The server uses voice recognition technology to convert the received voice data into text data. Google Cloud Speech-to-Text API is used here.

[0150] Input: Audio data added to the processing queue

[0151] Data processing: Speech recognition and text conversion with Google Cloud Speech-to-Text API

[0152] Output: Text data

[0153] Specific operation: The server sends audio data to the API and receives the returned text data.

[0154] Step 5: Analyzing the text data (server)

[0155] The server uses a generative AI model (e.g., OpenAI's GPT) to analyze the text data, using prompts to detect keywords and phrases characteristic of fraud.

[0156] Input: Text data

[0157] Data Processing: Text Analysis with Generative AI Models

[0158] Output: Fraud likelihood score

[0159] Example prompt sentence:

[0160] Analyze the following calls and rate them for potential fraud.

[0161] Call content: {text data goes here}

[0162] Specific operation: Insert text data into the prompt sentence and call the generative AI model to perform analysis.

[0163] Step 6: Scoring Fraud Potential (Server)

[0164] Based on the results of the generative AI model's analysis, a score is assigned to the likelihood of fraud. If the score exceeds a set threshold, it is deemed highly likely to be fraudulent.

[0165] Input: Analysis results of the generative AI model

[0166] Data processing: Calculating fraud likelihood scores

[0167] Output: Fraud probability determination result (True / False)

[0168] Specific operation: The generative AI model returns a score for the analysis results and checks whether the score exceeds a set threshold.

[0169] Step 7: Sending Notifications (Server)

[0170] If fraud is deemed likely, the server will notify the user and pre-defined third parties via text message, email, or dedicated in-app notification.

[0171] Input: Fraud possibility determination result

[0172] Data processing: generating and sending notification messages

[0173] Output: Users and third parties who received the notification

[0174] What happens: The server sends an alert to the user and their contacts via a pre-registered notification channel.

[0175] Step 8: Storing and Patterning Data (Server)

[0176] Calls that are deemed likely to be fraudulent are anonymized and stored in a fraud pattern database, which is used to improve future models and for fraud prevention education.

[0177] Input: Call content determined to be fraudulent

[0178] Data processing: anonymizing call content and storing it as patterned data

[0179] Output: Data stored in the fraud pattern database

[0180] Specific actions: Calls are anonymized and stored in a fraud pattern database, which is then periodically used to improve AI models and as training materials.

[0181] (Application example 1)

[0182] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0183] As telephone fraud techniques become more sophisticated, the risk of elderly people and general users becoming victims of fraud increases. To prevent such fraud, a system is needed that can analyze call content in real time, quickly detect potentially fraudulent calls, and notify users and their trusted contacts. However, existing systems lack the accuracy of call analysis and the speed of notification, making it difficult to completely prevent fraud.

[0184] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0185] In this invention, the server includes a means for recording the contents of the call, a means for analyzing the recorded contents of the call using a generative AI model, a means for determining the possibility of special fraud based on the analysis results, a means for sending a notification when there is a possibility of special fraud, a means for sending a notification to the user and trusted contacts when the analysis results exceed a certain threshold, and a means for anonymizing and saving the contents of the call for later use as learning data or educational material for fraud prevention. This enables early detection and prompt notification of call fraud, as well as data accumulation and learning for fraud prevention.

[0186] "Call content" refers to all audio information generated during a call.

[0187] "Means of recording" refers to the technology that captures the contents of a call in digital format and stores it as data.

[0188] A "generative AI model" refers to a program that analyzes data based on patterns and features that have been previously learned using machine learning algorithms.

[0189] "Means of analysis" refers to the ability to analyze the characteristics and patterns of input data using a generative AI model.

[0190] "Special fraud" refers to a method of defrauding unspecified people of money using the telephone or the Internet.

[0191] "Means for determining" refers to the function of evaluating whether or not a specific condition is met based on the analyzed results.

[0192] "Means for sending notifications" refers to communication technologies that allow the system to notify the user or designated contacts of specific information.

[0193] "Analysis results" refers to the information and evaluations obtained as a result of data analysis by a generative AI model.

[0194] A "certain threshold" refers to a reference value set based on the analysis results, and if this reference value is exceeded, a specific action will be taken.

[0195] "User" refers to the entity that uses the system, specifically, the individual or company that makes calls using the system.

[0196] "Trusted Contacts" refers to people or entities pre-defined by the user who are authorized to receive notifications regarding fraud and other emergencies.

[0197] "Anonymization" refers to techniques that remove or change personally identifiable information to protect the privacy of data.

[0198] "Storage" refers to techniques for storing data long-term and making it available for later reuse.

[0199] "Training data" refers to past data used as training material to improve the accuracy of generative AI models.

[0200] "Educational Materials" means written, audio, video, or other educational materials used to teach fraud prevention knowledge and techniques.

[0201] This invention is a system that records phone conversations, analyzes the recorded conversations using a generative AI model, determines the possibility of special fraud, and notifies the user. Specific embodiments for implementing this system are described in detail below.

[0202] System configuration

[0203] This system consists of a user's device, a server that stores and analyzes voice data, and a network for sending notifications.

[0204] Terminal

[0205] The device has the function of recording the contents of calls and sending the recorded data to a server in real time. When a call starts, the audio recording module automatically starts and buffers the audio data. This function is realized using a recording application on the smartphone or a dedicated call recording device.

[0206] server

[0207] The server receives the voice data sent from the device and analyzes it using a generative AI model. Specifically, a voice recognition module converts the voice data into text, and then analyzes the text data to determine whether it contains any signs of fraud. This analysis could use a machine learning algorithm (such as OpenAI's GPT-4 or Google BERT). Based on the analysis results, the server scores the likelihood of fraud, and if the score exceeds a certain threshold, it determines that there is a high possibility of fraud. In addition, the content of calls determined to be fraudulent is anonymized and stored as learning data or educational material.

[0208] Notification Network

[0209] The notification network notifies users and pre-defined trusted contacts when potential fraud is detected via text message, email, or dedicated in-app notifications, using the Twilio API or Firebase Cloud Messaging.

[0210] Specific examples

[0211] When user A receives a call from a self-proclaimed "bank representative" while on a call on his smartphone, the system operates as follows.

[0212] 1. When user A answers the phone, the device automatically records the call and sends the recording data to the server in real time.

[0213] 2. The server receives the voice data and the generative AI model analyzes it to confirm that the call contains keywords characteristic of fraud, such as "bank," "account number," and "transfer."

[0214] 3. The generative AI model generates a high fraud score (e.g., 93%). The server determines that there is a high probability of fraud and sends a notification.

[0215] 4. The server notifies User A and their trusted contacts via text message that a potentially fraudulent call was detected.

[0216] 5. The call transcripts are anonymized and stored in a fraud pattern database, which will be used to improve educational materials and AI models for future fraud prevention.

[0217] Example prompt sentence:

[0218] "Please convert the audio below to text and rate it as likely to be a scam. Please also be specific about which parts are characteristic of a scam.

[0219] Audio data: [Audio data URL]

[0220] This system enables early detection and rapid notification of toll fraud, as well as data accumulation and learning for fraud prevention.

[0221] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0222] Step 1:

[0223] A user starts a call on the device. The input is "start call." When the call starts, the device automatically starts the voice recording module and starts recording the contents of the call. The voice data is temporarily stored in a buffer. The output is the recorded "voice data of the call."

[0224] Step 2:

[0225] The device sends recorded audio data to the server in real time. The input is "call audio data." The device transfers the recorded file of the call content to the server via WebSocket or REST API. The output is "sent audio data" to the server.

[0226] Step 3:

[0227] The server receives the voice data and begins analysis using a generative AI model. The input is the "received voice data." The server converts the voice data into text using a speech recognition module. The text data is then input into a generative AI model (for example, OpenAI's GPT-4) for analysis. This detects whether the voice data contains features or patterns that indicate possible fraud. The output is the "analysis results."

[0228] Step 4:

[0229] The server scores the likelihood of fraud based on the analysis results. The input is the "analysis results." The generative AI model evaluates the likelihood of fraud based on prior learning and generates a score (e.g., a fraud score of 93%). The output is the "fraud score."

[0230] Step 5:

[0231] The server determines if the fraud score exceeds a certain threshold and prepares to send a notification if so. The inputs are the "fraud score" and the "threshold". In the case of a high fraud score above the threshold, the server generates a notification message and prepares to send it. The output is the "notification message".

[0232] Step 6:

[0233] The server sends notifications to the user and trusted contacts. The inputs are a "notification message" and "contact information." The server uses the Twilio API or Firebase Cloud Messaging to send notifications to inform the user and trusted contacts of possible fraud. The output is a "sent notification."

[0234] Step 7:

[0235] The server anonymizes the content of calls determined to be fraudulent and stores it in a pattern database. The input is "content of calls determined to be fraudulent." The server anonymizes the content of calls and stores it in a fraud pattern database for use as future learning data or educational material. The output is "stored anonymized data."

[0236] Step 8:

[0237] The server generates and distributes periodic reports to users and their families so that they can receive the latest information on fraud prevention. The inputs are "stored anonymized data" and "report distribution schedule." The server generates reports summarizing the latest fraud techniques and prevention strategies based on the stored data and distributes them periodically to users and trusted contacts. The output is "sent reports."

[0238] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0239] This invention is a system that records phone conversations, analyzes the recordings using a generative AI model and an emotion engine, determines the possibility of special fraud, and notifies the user if necessary. The following describes in detail an embodiment of this system.

[0240] System configuration

[0241] This system consists of the user's smartphone (device), a server that stores and analyzes voice data, a network for sending notifications, and an emotion engine that recognizes the user's emotions.

[0242] Terminal

[0243] The device has the function of recording the user's voice during a call and sending the data to the server in real time. When a call starts, the voice recording module automatically starts and buffers the voice data. The device is also equipped with a microphone and camera to capture the user's emotional data.

[0244] server

[0245] The server receives the voice and emotion data sent from the device and processes it in real time. The voice data is analyzed by a generative AI model to calculate a fraud probability score. The emotion engine also analyzes the user's emotional state.

[0246] Emotion Engine

[0247] The emotion engine is a module that recognizes the user's emotions from voice and image data. The emotion engine analyzes changes in voice tone and facial expressions to identify emotions such as nervousness, confusion, or surprise.

[0248] Notification Network

[0249] The notification network sends notifications to users and pre-defined trusted contacts (e.g., family members) when fraud is deemed likely, and can take emotional data into account to provide even more accurate warnings.

[0250] Program processing

[0251] Below we explain how each component of the system works.

[0252] Start recording a call (device)

[0253] When a user initiates a call on their smartphone, the device automatically records the call, storing the recording in a temporary buffer and sending it to a server in real time.

[0254] Receiving and analyzing voice data (server)

[0255] The server receives the voice data sent from the device and begins analysis using the generative AI model. The generative AI model converts the recorded data into text using a speech recognition module, and analyzes the text data to determine whether it contains any signs of fraud. As a result of the analysis, a score is assigned to the likelihood of fraud, and a final evaluation is made along with the user's emotional data using an emotion engine.

[0256] Acquisition and analysis of emotion data (device and server)

[0257] The device's microphone and camera capture the user's voice and facial expression data, which is then sent to a server that analyzes the data using an emotion engine to assess the user's emotional state. If the user expresses anxiety or confusion, this is also factored into the fraud likelihood score.

[0258] Possible fraud notification (server)

[0259] If a call is deemed likely to be fraudulent, the server sends a notification to pre-defined contacts (the user or one of their trusted contacts) informing them that a potentially fraudulent call has been detected and providing a warning based on the user's emotional state.

[0260] Storage and patterning of call content and emotion data (server)

[0261] If the call content and emotional data is determined to be potentially fraudulent, it is anonymized and stored in a database that records the characteristics of the fraud and the user's emotional state. This data will then be used to develop fraud prevention education materials and improve future AI models. Users and their families will receive regular updates on the latest fraud scams and prevention strategies.

[0262] Specific examples

[0263] System Operation Scenarios

[0264] For example, if user B receives a call from a self-proclaimed "police officer" while on a call on his smartphone, the system operates as follows.

[0265] 1. When User B answers the phone, the device automatically records the call and simultaneously captures emotion data.

[0266] 2. The recorded data and emotional data are sent to the server in real time.

[0267] 3. The generative AI model analyzes the voice data and detects fraud keywords such as "police" and "ID." In addition, the emotion engine detects User B's anxious voice and confused facial expression.

[0268] 4. Based on a high fraud score (e.g., 90%) and User B's emotional state, fraud is deemed more likely.

[0269] 5. The server sends a text message to User B and his family saying, "A likely fraudulent call has been detected. User B is in a state of anxiety."

[0270] 6. The call content and emotional data is anonymized and stored in a fraud pattern database.

[0271] 7. User B and his family will receive a report at a later date containing updated information on fraud prevention.

[0272] This system will enable more accurate fraud detection based on the user's emotional state, providing quick and accurate protection from fraud. As a result, it will provide an environment where all users, including the elderly, can make calls with peace of mind.

[0273] The processing flow will be explained below.

[0274] Step 1: Start recording the call (on your device)

[0275] When a user starts a call on the smartphone, the terminal detects a call start event.

[0276] The device will automatically start the audio recording module.

[0277] The device stores the recorded data in a temporary buffer and prepares it for transmission to the server in real time.

[0278] Step 2: Receiving audio data (server)

[0279] The server receives the voice data packets sent from the terminal.

[0280] The received audio data is stored in a temporary storage device.

[0281] The server queues the audio data for analysis by the generative AI model.

[0282] Step 3: Analyzing the voice data (server)

[0283] The server retrieves the audio data from the analysis queue.

[0284] The server uses voice recognition technology to convert the voice data into text data.

[0285] The server uses a generative AI model to analyze the text data and detect fraudulent features.

[0286] Based on the analysis results, a fraud likelihood score is calculated.

[0287] Step 4: Acquiring emotion data (device)

[0288] The device's microphone and camera capture the user's voice and facial expression data.

[0289] The terminal transmits audio and image data to the server in real time.

[0290] Step 5: Emotion data analysis (server)

[0291] The server uses an emotion engine to analyze the received audio and image data.

[0292] An emotion engine identifies the user's emotional state (e.g., anxiety, confusion, surprise, etc.).

[0293] The analysis results of the emotion data and voice data are combined to update the final fraud likelihood score.

[0294] Step 6: Determine the likelihood of fraud (server)

[0295] The server evaluates the updated fraud likelihood score.

[0296] Determine whether the score exceeds a set threshold.

[0297] If fraud is deemed likely, prepare to notify.

[0298] Step 7: Sending Notifications (Server)

[0299] The server generates a notification message if fraud is deemed likely.

[0300] The notification includes information about the likelihood of fraud and the user's emotional state.

[0301] Notifications will be sent to the user and pre-registered contacts (such as family members) via text message or in-app notification.

[0302] Step 8: Storing and patterning call content and emotion data (server)

[0303] The server anonymizes the content of calls that are deemed likely to be fraudulent.

[0304] The anonymized data, including emotional data, is stored in a fraud pattern database.

[0305] The latest fraud methods are analyzed based on the collected data.

[0306] Regularly provide fraud prevention educational information to you and your family.

[0307] Example 2

[0308] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0309] Conventional call recording systems have low detection accuracy for special frauds, making it difficult to prevent fraud victims. Furthermore, because they do not take into account the user's emotional state, they are unable to properly assess the anxiety and confusion felt by the user. Furthermore, when a potential fraud is detected, notification methods are limited, making it impossible to quickly communicate warnings to a wide range of contacts. These issues need to be resolved.

[0310] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0311] In this invention, the server includes a means for recording the content of the call, a means for analyzing the recorded content of the call using a generative AI model, a means for determining the possibility of special fraud based on the analysis results, a means for acquiring and analyzing emotional data, and a means for sending a notification if there is a possibility of special fraud. This allows for simultaneous analysis of the content of the call and the user's emotional state, enabling more accurate fraud detection and faster notification.

[0312] A "call recording device" is a device or software that allows a user to capture and save audio data in real time while making a call.

[0313] "Means of analysis using a generative AI model" refers to artificial intelligence technology that converts recorded audio data into text data and analyzes the characteristics of fraud.

[0314] "Means for determining the possibility of special fraud" refers to algorithms or programs for evaluating and scoring the possibility of fraud based on the analysis results of the generative AI model.

[0315] "Means for acquiring and analyzing emotional data" refers to emotion recognition technology for capturing a user's vocal tone and facial expressions and analyzing the data to assess the user's emotional state.

[0316] "Means for sending notifications" refers to communication means for sending information to the user and pre-defined trusted contacts if fraud is determined to be likely.

[0317] This invention is a system that records phone conversations, analyzes the recordings using a generative AI model and an emotion engine, determines the possibility of special fraud, and notifies the user if necessary. The following describes in detail an embodiment of this system.

[0318] System configuration

[0319] This system consists of the user's smartphone (device), a server that stores and analyzes voice data, a network for sending notifications, and an emotion engine that recognizes the user's emotions.

[0320] Terminal

[0321] The device has the function of recording the user's voice during a call and sending the data to the server in real time. When a call starts, the voice recording module automatically starts and buffers the voice data. The device is also equipped with a microphone and camera to capture the user's emotional data.

[0322] server

[0323] The server receives the voice and emotion data sent from the device and processes it in real time. The voice data is analyzed by a generative AI model to calculate a fraud probability score. The emotion engine also analyzes the user's emotional state.

[0324] Emotion Engine

[0325] The emotion engine is a module that recognizes the user's emotions from voice and image data. The emotion engine analyzes changes in voice tone and facial expressions to identify emotions such as nervousness, confusion, or surprise.

[0326] Notification Network

[0327] The notification network sends notifications to users and pre-defined trusted contacts (e.g., family members) when fraud is deemed likely, and can take emotional data into account to provide even more accurate warnings.

[0328] Program processing

[0329] Below we explain how each component of the system works.

[0330] Start recording a call (device)

[0331] When a user starts a call on their smartphone, the device automatically records the conversation. This recording is temporarily stored in a buffer and then sent to a server in real time. The device is also equipped with a microphone and camera to capture the user's emotional data.

[0332] Receiving and analyzing voice data (server)

[0333] The server receives the voice data sent from the device and begins analysis using the generative AI model. Specifically, it uses a voice recognition module to convert the recorded data into text data, and then analyzes the text data to determine whether it contains any fraudulent characteristics. This analysis calculates a fraud likelihood score. The emotion engine simultaneously analyzes the user's emotional data, and a final evaluation is made.

[0334] Acquisition and analysis of emotion data (device and server)

[0335] The device's microphone and camera capture the user's voice and facial expression data, which is then sent to a server that analyzes it using an emotion engine to assess the user's emotional state. If the user expresses anxiety or confusion, this is also factored into the fraud likelihood score.

[0336] Possible fraud notification (server)

[0337] If a fraudulent call is determined to be likely, the server sends a notification to a pre-defined contact with the message, "A fraudulent call has been detected. User B is in a state of anxiety."

[0338] Storage and patterning of call content and emotion data (server)

[0339] If a call is deemed potentially fraudulent, the call's content and emotional data will be stored in an anonymized database. This data will then be used to improve fraud prevention education materials and AI models. Users and their families will also receive regular updates on the latest fraud scams and prevention strategies.

[0340] Example: System Operation Scenarios

[0341] For example, if a user receives a call from a self-proclaimed "police officer" while on a call on their smartphone, the system operates as follows.

[0342] When the user answers the call, the device automatically records the call and simultaneously captures emotional data.

[0343] Recorded data and emotional data are sent to the server in real time.

[0344] A generative AI model analyzes the audio data to detect fraud keywords such as "police" and "ID," while an emotion engine detects anxious voices and confused facial expressions.

[0345] A high fraud score (e.g., 90%) and the user's emotional state are evaluated to determine the likelihood of fraud.

[0346] The server sends a text message to the user and their family saying, "A likely fraudulent call has been detected. The user is in a state of anxiety."

[0347] The call content and emotional data is anonymized and stored in a fraud pattern database.

[0348] Users and their families will receive a report at a later date containing updated information on fraud prevention.

[0349] Prompt Sentence Examples

[0350] "Specify how a system works to detect potential fraud while a user is on a call."

[0351] This system will enable more accurate fraud detection based on the user's emotional state, providing quick and accurate protection from fraud. As a result, it will provide an environment where all users, including the elderly, can make calls with peace of mind.

[0352] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0353] Step 1:

[0354] When a user initiates a call on their smartphone, the device automatically records the call, temporarily stores the recording in a buffer, and then transmits it to a server in real time.

[0355] Specific behavior:

[0356] 1. When a user makes or receives a call, the smartphone's voice recording module is automatically activated.

[0357] 2. Audio data is captured through the built-in microphone and temporarily stored in the device's memory.

[0358] 3. The buffered data is encrypted and sent to the server in real time.

[0359] Input: Call audio data

[0360] Output: Encrypted recording data

[0361] Step 2:

[0362] The server receives the voice data sent from the device and temporarily stores the recorded data in a database, while simultaneously starting analysis using the generative AI model.

[0363] Specific behavior:

[0364] 1. The server receives the encrypted voice data, decrypts it, and temporarily stores it in a database.

[0365] 2. The decoded voice data is input into the generative AI model and converted into text data using a voice recognition module.

[0366] 3. A generative AI model analyzes the text data and calculates a fraud likelihood score.

[0367] Input: Encrypted audio recording

[0368] Output: Analysis results including fraud likelihood score

[0369] Step 3:

[0370] The device's microphone and camera capture the user's voice and facial expression data, which is then sent to a server.

[0371] Specific behavior:

[0372] 1. The device captures the user's voice tone with the microphone.

[0373] 2. The camera captures the user's facial expressions in real time.

[0374] 3. The acquired audio and image data are sent to the server.

[0375] Input: Voice tone and facial expression data

[0376] Output: Voice tone and facial expression data sent to the server

[0377] Step 4:

[0378] The server receives the emotion data sent from the terminal and analyzes it using an emotion engine.

[0379] Specific behavior:

[0380] 1. The server inputs the received voice tone and facial expression data into the emotion engine.

[0381] 2. The emotion engine analyzes voice tone and facial expressions to assess the user's emotional state.

[0382] 3. The emotional state assessment results are combined with the deception likelihood score to form a final assessment.

[0383] Input: Voice tone and facial expression data

[0384] Output: Evaluation results including the user's emotional state

[0385] Step 5:

[0386] If fraud is deemed likely, the server will send a notification to pre-defined contacts.

[0387] Specific behavior:

[0388] 1. The server makes a comprehensive assessment based on the fraud probability score and the user's emotional state.

[0389] 2. Generate a notification message if fraud is determined to be likely.

[0390] 3. Send a notification message to you and your pre-defined trusted contacts.

[0391] Input: Evaluation results including the user's emotional state

[0392] Output: Notification message

[0393] Step 6:

[0394] The server anonymizes the content of calls that are deemed likely to be fraudulent and the user's emotional data and stores them in a database.

[0395] Specific behavior:

[0396] 1. Anonymize data that the analysis assesses as likely to be fraudulent.

[0397] 2. Store the anonymized data in a fraud pattern database.

[0398] 3. The data will later be used to improve fraud prevention educational materials and AI models.

[0399] Input: Call content and emotion data including evaluation results

[0400] Output: Anonymized database records

[0401] (Application example 2)

[0402] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0403] In today's society, the number of victims of telephone fraud is increasing, and elderly people in particular are prone to falling prey to the sophisticated tactics of fraudsters. Furthermore, conventional fraud prevention systems have issues with their difficulty in accurately identifying fraud and their inability to issue warnings at the appropriate time because they do not take the user's emotional state into account. This invention aims to provide a system that can accurately detect the possibility of fraud and issue warnings at the appropriate time by analyzing the content of phone calls and comprehensively assessing the user's emotional state.

[0404] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0405] In this invention, the server includes a means for recording call content, a means for analyzing the recorded call content with a generative AI model, a means for detecting fraud keywords, a means for acquiring and analyzing user emotion data, a means for scoring the likelihood of fraud based on the analysis results and the emotion data, and a means for sending a notification to the user and trusted contacts if the likelihood of fraud is high, thereby enabling advanced fraud detection and warning based not only on call content but also on the user's emotional state.

[0406] "Call content" refers to the audio and text data exchanged during a phone or video call.

[0407] "Recording" is the process of storing telephone conversations in digital or analog form.

[0408] A "generative AI model" is an artificial intelligence model that uses machine learning algorithms to analyze voice data and identify specific patterns and keywords.

[0409] "Analysis" is the process of evaluating recorded data to detect specific patterns or characteristics.

[0410] "Fraud keywords" are specific words or phrases used to characterize fraudulent activity.

[0411] "Emotion data" refers to data that includes changes in voice tone and facial expressions that indicate the emotional state of the user during a call.

[0412] "Scoring" is the process of evaluating data based on specific criteria and assigning them a score accordingly.

[0413] "Notification" is a communication method used to convey alerts or information to users or trusted contacts.

[0414] "Trusted Contacts" refers to people or organizations that a user pre-selects with whom they can share emergency or important information.

[0415] "Anonymization" is the process of removing personally identifiable information from data so that it cannot be identified by third parties.

[0416] A "pattern" refers to an arrangement or tendency of a set of data based on a particular characteristic or feature.

[0417] The system that realizes this invention is made up of a complex configuration including a user terminal, a server, a notification network, and an emotion engine.

[0418] Terminal

[0419] The device is assumed to be a smartphone used by the user. This smartphone is equipped with a voice recording module for recording the contents of calls. When a call is initiated, recording starts automatically and the voice data is sent to a server in real time. The device is also equipped with a microphone and a camera, which can capture the user's emotional data.

[0420] server

[0421] The server plays a central role in processing the voice and emotion data sent from the devices. The following software is installed on the server:

[0422] Generative AI model: The generative AI model uses a speech recognition module to convert the recording into text, then analyzes it for potential fraud. If fraud keywords are detected, the analysis results are scored.

[0423] Emotion Engine: The emotion engine analyzes voice tone and facial expression data to assess the user's emotional state. If the user is anxious or confused, this will also be reflected in the fraud likelihood score.

[0424] Notification Network

[0425] The server assesses the likelihood of fraud based on the results of the generative AI model and emotion engine analysis, and sends notifications to users and pre-defined trusted contacts if there is a high likelihood of fraud. The network can issue alerts via text message or in-app notification.

[0426] Specific examples

[0427] For example, if a user receives a phone call from someone who identifies themselves as a "police officer," the modules in the system will act as follows:

[0428] 1. When a user receives a call, the device's voice recording module automatically starts recording and transmits the voice data to the server in real time.

[0429] 2. A generative AI model is installed on the server, and fraud keywords such as "police" and "ID" are detected from the voice data.

[0430] 3. At the same time, the device's microphone and camera capture the user's voice tone and facial expression data, which are then analyzed by the emotion engine. It can be seen that the user's voice sounds anxious or their facial expression looks confused.

[0431] 4. Based on the analysis results of the generative AI model and the emotion engine, a high fraud score is calculated. Specifically, the generative AI model's analysis result indicates a fraud probability score of 90%, and the emotion engine's analysis result also determines that the user is in an anxious state.

[0432] 5. If the final evaluation results in a high fraud score (e.g., 90%), the server sends a notification to the user and trusted contacts stating, "A potentially fraudulent call has been detected."

[0433] An example prompt is:

[0434] "The user answers the phone and starts talking. At the same time as the voice data, facial expression data is also captured and sent to the server. Keywords such as 'police,' 'bank transfer,' and 'ID card' are detected on the server side, and at the same time the user's face shows signs of confusion or anxiety. The system rates these with a high score and notifies the user and their trusted contacts that 'A call with a high probability of fraud has been detected.'"

[0435] This allows users to be quickly warned of potential fraud and prevent it from happening. The system utilizes cutting-edge artificial intelligence and emotion recognition technology to provide an effective means of protecting users' safety in real time.

[0436] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0437] Step 1:

[0438] Start recording a call

[0439] When a user starts a call on their smartphone, the voice recording module is automatically started. The recorded voice data is temporarily stored in a buffer. In this step, the input is the call start trigger and the output is the recorded voice data.

[0440] Step 2:

[0441] Acquiring voice and emotion data

[0442] The microphone and camera on the device capture the user's voice tone and facial expression data. These data are sent to the server in real time in parallel. The input is voice and facial expression capture, and the output is voice data and emotional data.

[0443] Step 3:

[0444] Analysis of audio data

[0445] The server analyzes the received voice data using a generative AI model. The voice data is first converted into text, and fraud keywords are detected from the text data. The input is the voice data, and the output is the analyzed text data and fraud keywords.

[0446] Step 4:

[0447] Emotional Data Analysis

[0448] The server analyzes the received emotional data using an emotion engine. It evaluates the user's emotional state based on the voice tone and facial expression data. In this step, the input is the emotion data, and the output is the analyzed emotional state.

[0449] Step 5:

[0450] Fraud likelihood scoring

[0451] The server scores the likelihood of fraud based on the analysis results of the generative AI model and the analysis results of the emotional data by the emotion engine. The fraud keyword detection results are combined with the user's emotional state to calculate an overall score. The inputs are text data and emotional state, and the output is a fraud likelihood score.

[0452] Step 6:

[0453] Sending notifications

[0454] If the fraud likelihood score is high, the server will send a notification to the user and pre-defined trusted contacts. The notification will include a notification that a fraud likelihood call was detected and a warning based on the analysis results. The input is the fraud likelihood score and the output is the warning notification.

[0455] Step 7:

[0456] Call content and emotion data storage and patterning

[0457] The content and emotion data of calls that are judged to be highly likely to be fraudulent are anonymized and stored. This creates a database for future fraud prevention. The inputs are the call data and emotion data that are judged to be fraudulent, and the output is an updated fraud prevention pattern database.

[0458] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0459] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0460] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0461] [Second embodiment]

[0462] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0463] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0464] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0465] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0466] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0467] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0468] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0469] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0470] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0471] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0472] In the smart glasses 214, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0473] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0474] This invention is a system that records phone conversations, analyzes the recorded conversations using a generative AI model, determines the possibility of special fraud, and notifies the user. The following describes in detail an embodiment of this system.

[0475] System configuration

[0476] This system consists of the user's smartphone (terminal), a server that stores and analyzes voice data, and a network for sending notifications.

[0477] Terminal

[0478] The terminal has the function of recording voice data during a call and transmitting the data to the server in real time. When a call starts, the voice recording module is automatically started and the voice data is buffered.

[0479] server

[0480] The server receives the voice data sent from the device and processes it in real time. The generative AI model used here analyzes the voice data based on pre-trained fraud characteristics. Based on the analysis results, it scores the likelihood of fraud, and if the score exceeds a pre-set threshold, it is determined that there is a high probability of fraud. The server anonymizes and saves the content of calls that may be fraudulent, recording them as patterns.

[0481] Notification Network

[0482] The notification network sends notifications to users and pre-defined trusted contacts (e.g., family members) when potential fraud is detected via text message, email, or dedicated in-app notifications.

[0483] Program processing

[0484] Below we explain how each component of the system works.

[0485] Start recording a call (device)

[0486] When a user initiates a call on their smartphone, the device automatically records the call, storing the recording in a temporary buffer and sending it to a server in real time.

[0487] Receiving and analyzing voice data (server)

[0488] The server receives the voice data sent from the device and begins analysis using the generative AI model. The generative AI model converts the recorded data into text using a speech recognition module, and analyzes the text data to determine whether it contains any signs of fraud. As a result of the analysis, a score is assigned to the likelihood of fraud, and if the score is high, it is determined to be fraud.

[0489] Possible fraud notification (server)

[0490] If a call is deemed likely to be fraudulent, the server will send a notification to a pre-defined contact (either the user or one of their trusted contacts) stating that a potentially fraudulent call has been detected.

[0491] Save and patternize call content (server)

[0492] If a call is deemed potentially fraudulent, it is anonymized and stored in a pattern database that records the characteristics of fraud. This data will then be used to develop fraud prevention education materials and improve future AI models. Users and their families will receive regular updates on the latest fraud scams and prevention strategies.

[0493] Specific examples

[0494] System Operation Scenarios

[0495] For example, if user A receives a call from a self-proclaimed "bank representative" while on a call on his smartphone, the system operates as follows.

[0496] 1. When user A answers the phone, the device automatically records the call and sends the recording data to the server in real time.

[0497] 2. The server receives the audio data and the generative AI model analyzes it to ensure that the call contains typical scam keywords such as "bank," "account number," and "transfer."

[0498] 3. The generative AI model produces a high fraud score (e.g., 93%). The server determines that fraud is likely and prepares to send a notification.

[0499] 4. The server notifies User A and his / her family via text message that a potentially fraudulent call has been detected.

[0500] 5. The call content is anonymized and stored in a fraud pattern database.

[0501] 6. User A and his family will receive a report at a later date containing updated information on fraud prevention.

[0502] This system will quickly protect elderly people from fraud and enable them to take appropriate crime prevention measures, thereby providing an environment in which they can live their daily lives with peace of mind.

[0503] The processing flow will be explained below.

[0504] Step 1: Start recording the call (on your device)

[0505] When a user starts a call on the smartphone, the terminal detects a call start event.

[0506] The device will automatically start the audio recording module.

[0507] The device stores the recorded data in a temporary buffer and prepares it for transmission to the server in real time.

[0508] Step 2: Receiving audio data (server)

[0509] The server receives the voice data packets sent from the terminal.

[0510] The received audio data is stored in temporary storage.

[0511] The server queues the audio data for analysis by the generative AI model.

[0512] Step 3: Analyzing the voice data (server)

[0513] The server retrieves the audio data from the analysis queue.

[0514] The server uses voice recognition technology to convert the voice data into text data.

[0515] The server inputs the converted text data into the generative AI model.

[0516] The generative AI model analyzes the fraud features and calculates a fraud score.

[0517] Step 4: Determine the possibility of fraud (server)

[0518] The server evaluates the fraud scores returned by the generative AI model.

[0519] Determine whether the score exceeds a set threshold.

[0520] If fraud is deemed likely, prepare to notify.

[0521] Step 5: Sending Notifications (Server)

[0522] The server generates a notification message if fraud is deemed likely.

[0523] Notifications will be sent to the user and pre-registered contacts (family members, etc.).

[0524] Notifications will be sent via text message and in-app notifications.

[0525] Step 6: Sharing call content and patterning (server)

[0526] The server performs a process to anonymize the content of calls that are determined to be highly likely to be fraudulent.

[0527] Anonymized call transcripts are stored in a fraud pattern database.

[0528] The latest fraud methods are analyzed based on the collected data.

[0529] Regularly provide fraud prevention educational information to you and your family.

[0530] Example 1

[0531] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0532] In recent years, the methods used by special fraudsters targeting the elderly have become more sophisticated, resulting in an increase in the number of victims. To deal with such situations, it is necessary to analyze call content in real time and quickly identify dangerous calls. However, with conventional methods, recording and analyzing call content is time-consuming and laborious, making it difficult to respond quickly. In addition, there is a lack of systems that can effectively identify possible fraud and notify users in a reliable manner. To solve these issues, it is necessary to provide a system that can perform an integrated process from recording call content to detecting fraud, notifying users, and storing data.

[0533] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0534] In this invention, the server includes means for automatically recording the contents of a call when the user initiates a call, means for transmitting the recorded voice data to the server in real time, means for converting the received voice data into text data using voice recognition technology, means for analyzing the converted text data using a generative AI model to determine the possibility of fraud, means for sending a notification to the user and a designated third party if there is a possibility of fraud, and means for anonymizing and saving the contents of a call that is likely to be fraudulent and recording it in a fraud pattern database, thereby enabling real-time analysis of the contents of the call and rapid detection of fraud.

[0535] The "means for automatically recording telephone conversations" refers to a device or software that has the function of automatically recording telephone conversations when a user initiates a call and temporarily storing the recorded conversations in a buffer.

[0536] "Means for transmitting recorded voice data to a server in real time" refers to a device or software that has the function of transmitting recorded voice data to a server in real time at regular intervals or for each data size.

[0537] "Means for converting received voice data into text data using voice recognition technology" refers to a device or software that includes a voice recognition module or API for analyzing received voice data and converting its contents into text.

[0538] A "means for analyzing text data converted using a generative AI model to determine the likelihood of fraud" is a device or software that includes algorithms or methods for analyzing text data using generative AI to identify characteristics of fraud.

[0539] "Means for sending notifications to the user and designated third parties in the event of possible fraud" refers to a device or software that has the functionality to send notifications to pre-defined contacts in the event that it is determined that there is a high possibility of fraud.

[0540] "Means for anonymizing and storing phone call content that is likely to be fraudulent and recording it in a fraud pattern database" refers to a device or software that has the function of anonymizing phone call content that is determined to be likely to be fraudulent and storing the characteristics of such phone call in a database.

[0541] This invention is a system that records phone conversations, analyzes the recorded conversations using a generative AI model, determines the possibility of special fraud, and notifies the user. Specific embodiments for implementing this system are described in detail below.

[0542] System configuration

[0543] This system consists of a user's communication device (e.g., a smartphone), a server that stores and analyzes voice data, and a communication network for sending notifications.

[0544] communication terminal

[0545] When a user starts a call, the communication device automatically records the call. The recorded voice data is sent to the server in real time. The device is equipped with a voice recognition module, which automatically buffers the voice data when a call starts.

[0546] server

[0547] The server receives and processes the voice data sent from the device. The server is equipped with speech recognition technology using the Google Cloud Speech-to-Text API, which converts the voice data into text data. The converted text data is analyzed by OpenAI's generative AI model (e.g., the GPT series). This generative AI model detects keywords and phrases characteristic of fraud and scores the likelihood of fraud. If the score exceeds a set threshold, the call is deemed to have a high probability of fraud. The content of calls determined to be fraudulent is anonymized and stored in a fraud pattern database.

[0548] Notification Network

[0549] The notification network will notify users and pre-defined trusted contacts (e.g., family members) about potential fraud via text message, email, or dedicated in-app notifications.

[0550] Specific examples

[0551] System Operation Scenarios

[0552] For example, if user A receives a call from someone claiming to be a "bank representative" while on a call on his smartphone, the system operates as follows:

[0553] 1. When user A answers the phone, the device automatically records the call and sends the recording data to the server in real time.

[0554] 2. The server receives the audio data and converts it into text using the Google Cloud Speech-to-Text API.

[0555] 3. The text data is analyzed using OpenAI's generative AI model (e.g., the GPT series). For example, it checks whether the call content contains keywords such as "bank," "account number," or "transfer."

[0556] 4. The generative AI model generates a high fraud score based on the analysis results. For example, if the fraud score is evaluated as 93%, the server determines that there is a high probability of fraud and prepares to send a notification.

[0557] 5. The server notifies User A and his / her family via text message that a potentially fraudulent call has been detected.

[0558] 6. The call content is anonymized and stored in a fraud pattern database.

[0559] 7. User A and his family will receive a report at a later date containing updated information on fraud prevention.

[0560] Example prompt for a generative AI model:

[0561] Analyze calls and score them for potential fraud.

[0562] Caller details: "Hello, I'm a bank representative. What's your account number?"

[0563] This system will quickly protect elderly people from fraud and enable them to take appropriate crime prevention measures, thereby providing an environment in which they can live their daily lives with peace of mind.

[0564] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0565] Step 1: Start recording the call (on your device)

[0566] When a user starts a call on their smartphone, the device's recording module automatically starts and records the call. The recording data is temporarily stored in a buffer.

[0567] Input: Call start signal

[0568] Data processing: The recording module buffers the call audio

[0569] Output: Recorded audio data

[0570] Specific behavior: When the user presses the call button in the phone app, the recording module starts in the background and begins recording the call audio.

[0571] Step 2: Sending audio data (terminal)

[0572] The recorded audio data is sent to the server in real time using a secure encryption protocol (e.g., TLS).

[0573] Input: Buffered audio data

[0574] Data processing: Transmitted to the server in real time (encrypted)

[0575] Output: Audio data sent to the server

[0576] Specific operation: Every time the recording data reaches a certain size (e.g., 5 seconds), the device sends this data to the server.

[0577] Step 3: Receiving audio data (server)

[0578] The server receives the voice data sent from the terminal and adds it to the processing queue. The received data is encrypted to ensure security.

[0579] Input: Audio data sent from the device

[0580] Data processing: Decode received data and add it to the processing queue

[0581] Output: Audio data added to the processing queue

[0582] Specific operation: The server analyzes the arriving data packet, decodes it, and then queues it for processing.

[0583] Step 4: Converting audio data to text (server)

[0584] The server uses voice recognition technology to convert the received voice data into text data. Google Cloud Speech-to-Text API is used here.

[0585] Input: Audio data added to the processing queue

[0586] Data processing: Speech recognition and text conversion with Google Cloud Speech-to-Text API

[0587] Output: Text data

[0588] Specific operation: The server sends audio data to the API and receives the returned text data.

[0589] Step 5: Analyzing the text data (server)

[0590] The server uses a generative AI model (e.g., OpenAI's GPT) to analyze the text data, using prompts to detect keywords and phrases characteristic of fraud.

[0591] Input: Text data

[0592] Data Processing: Text Analysis with Generative AI Models

[0593] Output: Fraud likelihood score

[0594] Example prompt sentence:

[0595] Analyze the following calls and rate them for potential fraud.

[0596] Call content: {text data goes here}

[0597] Specific operation: Insert text data into the prompt sentence and call the generative AI model to perform analysis.

[0598] Step 6: Scoring Fraud Potential (Server)

[0599] Based on the results of the generative AI model's analysis, a score is assigned to the likelihood of fraud. If the score exceeds a set threshold, it is deemed highly likely to be fraudulent.

[0600] Input: Analysis results of the generative AI model

[0601] Data processing: Calculating fraud likelihood scores

[0602] Output: Fraud probability determination result (True / False)

[0603] Specific operation: The generative AI model returns a score for the analysis results and checks whether the score exceeds a set threshold.

[0604] Step 7: Sending Notifications (Server)

[0605] If fraud is deemed likely, the server will notify the user and pre-defined third parties via text message, email, or dedicated in-app notification.

[0606] Input: Fraud possibility determination result

[0607] Data processing: generating and sending notification messages

[0608] Output: Users and third parties who received the notification

[0609] What happens: The server sends an alert to the user and their contacts via a pre-registered notification channel.

[0610] Step 8: Storing and Patterning Data (Server)

[0611] Calls that are deemed likely to be fraudulent are anonymized and stored in a fraud pattern database, which is used to improve future models and for fraud prevention education.

[0612] Input: Call content determined to be fraudulent

[0613] Data processing: anonymizing call content and storing it as patterned data

[0614] Output: Data stored in the fraud pattern database

[0615] Specific actions: Calls are anonymized and stored in a fraud pattern database, which is then periodically used to improve AI models and as training materials.

[0616] (Application example 1)

[0617] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0618] As telephone fraud techniques become more sophisticated, the risk of elderly people and general users becoming victims of fraud increases. To prevent such fraud, a system is needed that can analyze call content in real time, quickly detect potentially fraudulent calls, and notify users and their trusted contacts. However, existing systems lack the accuracy of call analysis and the speed of notification, making it difficult to completely prevent fraud.

[0619] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0620] In this invention, the server includes a means for recording the contents of the call, a means for analyzing the recorded contents of the call using a generative AI model, a means for determining the possibility of special fraud based on the analysis results, a means for sending a notification when there is a possibility of special fraud, a means for sending a notification to the user and trusted contacts when the analysis results exceed a certain threshold, and a means for anonymizing and saving the contents of the call for later use as learning data or educational material for fraud prevention. This enables early detection and prompt notification of call fraud, as well as data accumulation and learning for fraud prevention.

[0621] "Call content" refers to all audio information generated during a call.

[0622] "Means of recording" refers to the technology that captures the contents of a call in digital format and stores it as data.

[0623] A "generative AI model" refers to a program that analyzes data based on patterns and features that have been previously learned using machine learning algorithms.

[0624] "Means of analysis" refers to the ability to analyze the characteristics and patterns of input data using a generative AI model.

[0625] "Special fraud" refers to a method of defrauding unspecified people of money using the telephone or the Internet.

[0626] "Means for determining" refers to the function of evaluating whether or not a specific condition is met based on the analyzed results.

[0627] "Means for sending notifications" refers to communication technologies that allow the system to notify the user or designated contacts of specific information.

[0628] "Analysis results" refers to the information and evaluations obtained as a result of data analysis by a generative AI model.

[0629] A "certain threshold" refers to a reference value set based on the analysis results, and if this reference value is exceeded, a specific action will be taken.

[0630] "User" refers to the entity that uses the system, specifically, the individual or company that makes calls using the system.

[0631] "Trusted Contacts" refers to people or entities pre-defined by the user who are authorized to receive notifications regarding fraud and other emergencies.

[0632] "Anonymization" refers to techniques that remove or change personally identifiable information to protect the privacy of data.

[0633] "Storage" refers to techniques for storing data long-term and making it available for later reuse.

[0634] "Training data" refers to past data used as training material to improve the accuracy of generative AI models.

[0635] "Educational Materials" means written, audio, video, or other educational materials used to teach fraud prevention knowledge and techniques.

[0636] This invention is a system that records phone conversations, analyzes the recorded conversations using a generative AI model, determines the possibility of special fraud, and notifies the user. Specific embodiments for implementing this system are described in detail below.

[0637] System configuration

[0638] This system consists of a user's device, a server that stores and analyzes voice data, and a network for sending notifications.

[0639] Terminal

[0640] The device has the function of recording the contents of calls and sending the recorded data to a server in real time. When a call starts, the audio recording module automatically starts and buffers the audio data. This function is realized using a recording application on the smartphone or a dedicated call recording device.

[0641] server

[0642] The server receives the voice data sent from the device and analyzes it using a generative AI model. Specifically, a voice recognition module converts the voice data into text, and then analyzes the text data to determine whether it contains any signs of fraud. This analysis could use a machine learning algorithm (such as OpenAI's GPT-4 or Google BERT). Based on the analysis results, the server scores the likelihood of fraud, and if the score exceeds a certain threshold, it determines that there is a high possibility of fraud. In addition, the content of calls determined to be fraudulent is anonymized and stored as learning data or educational material.

[0643] Notification Network

[0644] The notification network notifies users and pre-defined trusted contacts when potential fraud is detected via text message, email, or dedicated in-app notifications, using the Twilio API or Firebase Cloud Messaging.

[0645] Specific examples

[0646] When user A receives a call from a self-proclaimed "bank representative" while on a call on his smartphone, the system operates as follows.

[0647] 1. When user A answers the phone, the device automatically records the call and sends the recording data to the server in real time.

[0648] 2. The server receives the voice data and the generative AI model analyzes it to confirm that the call contains keywords characteristic of fraud, such as "bank," "account number," and "transfer."

[0649] 3. The generative AI model generates a high fraud score (e.g., 93%). The server determines that there is a high probability of fraud and sends a notification.

[0650] 4. The server notifies User A and their trusted contacts via text message that a potentially fraudulent call was detected.

[0651] 5. The call transcripts are anonymized and stored in a fraud pattern database, which will be used to improve educational materials and AI models for future fraud prevention.

[0652] Example prompt sentence:

[0653] "Please convert the audio below to text and rate it as likely to be a scam. Please also be specific about which parts are characteristic of a scam.

[0654] Audio data: [Audio data URL]

[0655] This system enables early detection and rapid notification of toll fraud, as well as data accumulation and learning for fraud prevention.

[0656] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0657] Step 1:

[0658] A user starts a call on the device. The input is "start call." When the call starts, the device automatically starts the voice recording module and starts recording the contents of the call. The voice data is temporarily stored in a buffer. The output is the recorded "voice data of the call."

[0659] Step 2:

[0660] The device sends recorded audio data to the server in real time. The input is "call audio data." The device transfers the recorded file of the call content to the server via WebSocket or REST API. The output is "sent audio data" to the server.

[0661] Step 3:

[0662] The server receives the voice data and begins analysis using a generative AI model. The input is the "received voice data." The server converts the voice data into text using a speech recognition module. The text data is then input into a generative AI model (for example, OpenAI's GPT-4) for analysis. This detects whether the voice data contains features or patterns that indicate possible fraud. The output is the "analysis results."

[0663] Step 4:

[0664] The server scores the likelihood of fraud based on the analysis results. The input is the "analysis results." The generative AI model evaluates the likelihood of fraud based on prior learning and generates a score (e.g., a fraud score of 93%). The output is the "fraud score."

[0665] Step 5:

[0666] The server determines if the fraud score exceeds a certain threshold and prepares to send a notification if so. The inputs are the "fraud score" and the "threshold". In the case of a high fraud score above the threshold, the server generates a notification message and prepares to send it. The output is the "notification message".

[0667] Step 6:

[0668] The server sends notifications to the user and trusted contacts. The inputs are a "notification message" and "contact information." The server uses the Twilio API or Firebase Cloud Messaging to send notifications to inform the user and trusted contacts of possible fraud. The output is a "sent notification."

[0669] Step 7:

[0670] The server anonymizes the content of calls determined to be fraudulent and stores it in a pattern database. The input is "content of calls determined to be fraudulent." The server anonymizes the content of calls and stores it in a fraud pattern database for use as future learning data or educational material. The output is "stored anonymized data."

[0671] Step 8:

[0672] The server generates and distributes periodic reports to users and their families so that they can receive the latest information on fraud prevention. The inputs are "stored anonymized data" and "report distribution schedule." The server generates reports summarizing the latest fraud techniques and prevention strategies based on the stored data and distributes them periodically to users and trusted contacts. The output is "sent reports."

[0673] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0674] This invention is a system that records phone conversations, analyzes the recordings using a generative AI model and an emotion engine, determines the possibility of special fraud, and notifies the user if necessary. The following describes in detail an embodiment of this system.

[0675] System configuration

[0676] This system consists of the user's smartphone (device), a server that stores and analyzes voice data, a network for sending notifications, and an emotion engine that recognizes the user's emotions.

[0677] Terminal

[0678] The device has the function of recording the user's voice during a call and sending the data to the server in real time. When a call starts, the voice recording module automatically starts and buffers the voice data. The device is also equipped with a microphone and camera to capture the user's emotional data.

[0679] server

[0680] The server receives the voice and emotion data sent from the device and processes it in real time. The voice data is analyzed by a generative AI model to calculate a fraud probability score. The emotion engine also analyzes the user's emotional state.

[0681] Emotion Engine

[0682] The emotion engine is a module that recognizes the user's emotions from voice and image data. The emotion engine analyzes changes in voice tone and facial expressions to identify emotions such as nervousness, confusion, or surprise.

[0683] Notification Network

[0684] The notification network sends notifications to users and pre-defined trusted contacts (e.g., family members) when fraud is deemed likely, and can take emotional data into account to provide even more accurate warnings.

[0685] Program processing

[0686] Below we explain how each component of the system works.

[0687] Start recording a call (device)

[0688] When a user initiates a call on their smartphone, the device automatically records the call, storing the recording in a temporary buffer and sending it to a server in real time.

[0689] Receiving and analyzing voice data (server)

[0690] The server receives the voice data sent from the device and begins analysis using the generative AI model. The generative AI model converts the recorded data into text using a speech recognition module, and analyzes the text data to determine whether it contains any signs of fraud. As a result of the analysis, a score is assigned to the likelihood of fraud, and a final evaluation is made along with the user's emotional data using an emotion engine.

[0691] Acquisition and analysis of emotion data (device and server)

[0692] The device's microphone and camera capture the user's voice and facial expression data, which is then sent to a server that analyzes the data using an emotion engine to assess the user's emotional state. If the user expresses anxiety or confusion, this is also factored into the fraud likelihood score.

[0693] Possible fraud notification (server)

[0694] If a call is deemed likely to be fraudulent, the server sends a notification to pre-defined contacts (the user or one of their trusted contacts) informing them that a potentially fraudulent call has been detected and providing a warning based on the user's emotional state.

[0695] Storage and patterning of call content and emotion data (server)

[0696] If the call content and emotional data is determined to be potentially fraudulent, it is anonymized and stored in a database that records the characteristics of the fraud and the user's emotional state. This data will then be used to develop fraud prevention education materials and improve future AI models. Users and their families will receive regular updates on the latest fraud scams and prevention strategies.

[0697] Specific examples

[0698] System Operation Scenarios

[0699] For example, if user B receives a call from a self-proclaimed "police officer" while on a call on his smartphone, the system operates as follows.

[0700] 1. When User B answers the phone, the device automatically records the call and simultaneously captures emotion data.

[0701] 2. The recorded data and emotional data are sent to the server in real time.

[0702] 3. The generative AI model analyzes the voice data and detects fraud keywords such as "police" and "ID." In addition, the emotion engine detects User B's anxious voice and confused facial expression.

[0703] 4. Based on a high fraud score (e.g., 90%) and User B's emotional state, fraud is deemed more likely.

[0704] 5. The server sends a text message to User B and his family saying, "A likely fraudulent call has been detected. User B is in a state of anxiety."

[0705] 6. The call content and emotional data is anonymized and stored in a fraud pattern database.

[0706] 7. User B and his family will receive a report at a later date containing updated information on fraud prevention.

[0707] This system will enable more accurate fraud detection based on the user's emotional state, providing quick and accurate protection from fraud. As a result, it will provide an environment where all users, including the elderly, can make calls with peace of mind.

[0708] The processing flow will be explained below.

[0709] Step 1: Start recording the call (on your device)

[0710] When a user starts a call on the smartphone, the terminal detects a call start event.

[0711] The device will automatically start the audio recording module.

[0712] The device stores the recorded data in a temporary buffer and prepares it for transmission to the server in real time.

[0713] Step 2: Receiving audio data (server)

[0714] The server receives the voice data packets sent from the terminal.

[0715] The received audio data is stored in a temporary storage device.

[0716] The server queues the audio data for analysis by the generative AI model.

[0717] Step 3: Analyzing the voice data (server)

[0718] The server retrieves the audio data from the analysis queue.

[0719] The server uses voice recognition technology to convert the voice data into text data.

[0720] The server uses a generative AI model to analyze the text data and detect fraudulent features.

[0721] Based on the analysis results, a fraud likelihood score is calculated.

[0722] Step 4: Acquiring emotion data (device)

[0723] The device's microphone and camera capture the user's voice and facial expression data.

[0724] The terminal transmits audio and image data to the server in real time.

[0725] Step 5: Emotion data analysis (server)

[0726] The server uses an emotion engine to analyze the received audio and image data.

[0727] An emotion engine identifies the user's emotional state (e.g., anxiety, confusion, surprise, etc.).

[0728] The analysis results of the emotion data and voice data are combined to update the final fraud likelihood score.

[0729] Step 6: Determine the likelihood of fraud (server)

[0730] The server evaluates the updated fraud likelihood score.

[0731] Determine whether the score exceeds a set threshold.

[0732] If fraud is deemed likely, prepare to notify.

[0733] Step 7: Sending Notifications (Server)

[0734] The server generates a notification message if fraud is deemed likely.

[0735] The notification includes information about the likelihood of fraud and the user's emotional state.

[0736] Notifications will be sent to the user and pre-registered contacts (such as family members) via text message or in-app notification.

[0737] Step 8: Storing and patterning call content and emotion data (server)

[0738] The server anonymizes the content of calls that are deemed likely to be fraudulent.

[0739] The anonymized data, including emotional data, is stored in a fraud pattern database.

[0740] The latest fraud methods are analyzed based on the collected data.

[0741] Regularly provide fraud prevention educational information to you and your family.

[0742] Example 2

[0743] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0744] Conventional call recording systems have low detection accuracy for special frauds, making it difficult to prevent fraud victims. Furthermore, because they do not take into account the user's emotional state, they are unable to properly assess the anxiety and confusion felt by the user. Furthermore, when a potential fraud is detected, notification methods are limited, making it impossible to quickly communicate warnings to a wide range of contacts. These issues need to be resolved.

[0745] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0746] In this invention, the server includes a means for recording the content of the call, a means for analyzing the recorded content of the call using a generative AI model, a means for determining the possibility of special fraud based on the analysis results, a means for acquiring and analyzing emotional data, and a means for sending a notification if there is a possibility of special fraud. This allows for simultaneous analysis of the content of the call and the user's emotional state, enabling more accurate fraud detection and faster notification.

[0747] A "call recording device" is a device or software that allows a user to capture and save audio data in real time while making a call.

[0748] "Means of analysis using a generative AI model" refers to artificial intelligence technology that converts recorded audio data into text data and analyzes the characteristics of fraud.

[0749] "Means for determining the possibility of special fraud" refers to algorithms or programs for evaluating and scoring the possibility of fraud based on the analysis results of the generative AI model.

[0750] "Means for acquiring and analyzing emotional data" refers to emotion recognition technology for capturing a user's vocal tone and facial expressions and analyzing the data to assess the user's emotional state.

[0751] "Means for sending notifications" refers to communication means for sending information to the user and pre-defined trusted contacts if fraud is determined to be likely.

[0752] This invention is a system that records phone conversations, analyzes the recordings using a generative AI model and an emotion engine, determines the possibility of special fraud, and notifies the user if necessary. The following describes in detail an embodiment of this system.

[0753] System configuration

[0754] This system consists of the user's smartphone (device), a server that stores and analyzes voice data, a network for sending notifications, and an emotion engine that recognizes the user's emotions.

[0755] Terminal

[0756] The device has the function of recording the user's voice during a call and sending the data to the server in real time. When a call starts, the voice recording module automatically starts and buffers the voice data. The device is also equipped with a microphone and camera to capture the user's emotional data.

[0757] server

[0758] The server receives the voice and emotion data sent from the device and processes it in real time. The voice data is analyzed by a generative AI model to calculate a fraud probability score. The emotion engine also analyzes the user's emotional state.

[0759] Emotion Engine

[0760] The emotion engine is a module that recognizes the user's emotions from voice and image data. The emotion engine analyzes changes in voice tone and facial expressions to identify emotions such as nervousness, confusion, or surprise.

[0761] Notification Network

[0762] The notification network sends notifications to users and pre-defined trusted contacts (e.g., family members) when fraud is deemed likely, and can take emotional data into account to provide even more accurate warnings.

[0763] Program processing

[0764] Below we explain how each component of the system works.

[0765] Start recording a call (device)

[0766] When a user starts a call on their smartphone, the device automatically records the conversation. This recording is temporarily stored in a buffer and then sent to a server in real time. The device is also equipped with a microphone and camera to capture the user's emotional data.

[0767] Receiving and analyzing voice data (server)

[0768] The server receives the voice data sent from the device and begins analysis using the generative AI model. Specifically, it uses a voice recognition module to convert the recorded data into text data, and then analyzes the text data to determine whether it contains any fraudulent characteristics. This analysis calculates a fraud likelihood score. The emotion engine simultaneously analyzes the user's emotional data, and a final evaluation is made.

[0769] Acquisition and analysis of emotion data (device and server)

[0770] The device's microphone and camera capture the user's voice and facial expression data, which is then sent to a server that analyzes it using an emotion engine to assess the user's emotional state. If the user expresses anxiety or confusion, this is also factored into the fraud likelihood score.

[0771] Possible fraud notification (server)

[0772] If a fraudulent call is determined to be likely, the server sends a notification to a pre-defined contact with the message, "A fraudulent call has been detected. User B is in a state of anxiety."

[0773] Storage and patterning of call content and emotion data (server)

[0774] If a call is deemed potentially fraudulent, the call's content and emotional data will be stored in an anonymized database. This data will then be used to improve fraud prevention education materials and AI models. Users and their families will also receive regular updates on the latest fraud scams and prevention strategies.

[0775] Example: System Operation Scenarios

[0776] For example, if a user receives a call from a self-proclaimed "police officer" while on a call on their smartphone, the system operates as follows.

[0777] When the user answers the call, the device automatically records the call and simultaneously captures emotional data.

[0778] Recorded data and emotional data are sent to the server in real time.

[0779] A generative AI model analyzes the audio data to detect fraud keywords such as "police" and "ID," while an emotion engine detects anxious voices and confused facial expressions.

[0780] A high fraud score (e.g., 90%) and the user's emotional state are evaluated to determine the likelihood of fraud.

[0781] The server sends a text message to the user and their family saying, "A likely fraudulent call has been detected. The user is in a state of anxiety."

[0782] The call content and emotional data is anonymized and stored in a fraud pattern database.

[0783] Users and their families will receive a report at a later date containing updated information on fraud prevention.

[0784] Prompt Sentence Examples

[0785] "Specify how a system works to detect potential fraud while a user is on a call."

[0786] This system will enable more accurate fraud detection based on the user's emotional state, providing quick and accurate protection from fraud. As a result, it will provide an environment where all users, including the elderly, can make calls with peace of mind.

[0787] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0788] Step 1:

[0789] When a user initiates a call on their smartphone, the device automatically records the call, temporarily stores the recording in a buffer, and then transmits it to a server in real time.

[0790] Specific behavior:

[0791] 1. When a user makes or receives a call, the smartphone's voice recording module is automatically activated.

[0792] 2. Audio data is captured through the built-in microphone and temporarily stored in the device's memory.

[0793] 3. The buffered data is encrypted and sent to the server in real time.

[0794] Input: Call audio data

[0795] Output: Encrypted recording data

[0796] Step 2:

[0797] The server receives the voice data sent from the device and temporarily stores the recorded data in a database, while simultaneously starting analysis using the generative AI model.

[0798] Specific behavior:

[0799] 1. The server receives the encrypted voice data, decrypts it, and temporarily stores it in a database.

[0800] 2. The decoded voice data is input into the generative AI model and converted into text data using a voice recognition module.

[0801] 3. A generative AI model analyzes the text data and calculates a fraud likelihood score.

[0802] Input: Encrypted audio recording

[0803] Output: Analysis results including fraud likelihood score

[0804] Step 3:

[0805] The device's microphone and camera capture the user's voice and facial expression data, which is then sent to a server.

[0806] Specific behavior:

[0807] 1. The device captures the user's voice tone with the microphone.

[0808] 2. The camera captures the user's facial expressions in real time.

[0809] 3. The acquired audio and image data are sent to the server.

[0810] Input: Voice tone and facial expression data

[0811] Output: Voice tone and facial expression data sent to the server

[0812] Step 4:

[0813] The server receives the emotion data sent from the terminal and analyzes it using an emotion engine.

[0814] Specific behavior:

[0815] 1. The server inputs the received voice tone and facial expression data into the emotion engine.

[0816] 2. The emotion engine analyzes voice tone and facial expressions to assess the user's emotional state.

[0817] 3. The emotional state assessment results are combined with the deception likelihood score to form a final assessment.

[0818] Input: Voice tone and facial expression data

[0819] Output: Evaluation results including the user's emotional state

[0820] Step 5:

[0821] If fraud is deemed likely, the server will send a notification to pre-defined contacts.

[0822] Specific behavior:

[0823] 1. The server makes a comprehensive assessment based on the fraud probability score and the user's emotional state.

[0824] 2. Generate a notification message if fraud is determined to be likely.

[0825] 3. Send a notification message to you and your pre-defined trusted contacts.

[0826] Input: Evaluation results including the user's emotional state

[0827] Output: Notification message

[0828] Step 6:

[0829] The server anonymizes the content of calls that are deemed likely to be fraudulent and the user's emotional data and stores them in a database.

[0830] Specific behavior:

[0831] 1. Anonymize data that the analysis assesses as likely to be fraudulent.

[0832] 2. Store the anonymized data in a fraud pattern database.

[0833] 3. The data will later be used to improve fraud prevention educational materials and AI models.

[0834] Input: Call content and emotion data including evaluation results

[0835] Output: Anonymized database records

[0836] (Application example 2)

[0837] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0838] In today's society, the number of victims of telephone fraud is increasing, and elderly people in particular are prone to falling prey to the sophisticated tactics of fraudsters. Furthermore, conventional fraud prevention systems have issues with their difficulty in accurately identifying fraud and their inability to issue warnings at the appropriate time because they do not take the user's emotional state into account. This invention aims to provide a system that can accurately detect the possibility of fraud and issue warnings at the appropriate time by analyzing the content of phone calls and comprehensively assessing the user's emotional state.

[0839] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0840] In this invention, the server includes a means for recording call content, a means for analyzing the recorded call content with a generative AI model, a means for detecting fraud keywords, a means for acquiring and analyzing user emotion data, a means for scoring the likelihood of fraud based on the analysis results and the emotion data, and a means for sending a notification to the user and trusted contacts if the likelihood of fraud is high, thereby enabling advanced fraud detection and warning based not only on call content but also on the user's emotional state.

[0841] "Call content" refers to the audio and text data exchanged during a phone or video call.

[0842] "Recording" is the process of storing telephone conversations in digital or analog form.

[0843] A "generative AI model" is an artificial intelligence model that uses machine learning algorithms to analyze voice data and identify specific patterns and keywords.

[0844] "Analysis" is the process of evaluating recorded data to detect specific patterns or characteristics.

[0845] "Fraud keywords" are specific words or phrases used to characterize fraudulent activity.

[0846] "Emotion data" refers to data that includes changes in voice tone and facial expressions that indicate the emotional state of the user during a call.

[0847] "Scoring" is the process of evaluating data based on specific criteria and assigning them a score accordingly.

[0848] "Notification" is a communication method used to convey alerts or information to users or trusted contacts.

[0849] "Trusted Contacts" refers to people or organizations that a user pre-selects with whom they can share emergency or important information.

[0850] "Anonymization" is the process of removing personally identifiable information from data so that it cannot be identified by third parties.

[0851] A "pattern" refers to an arrangement or tendency of a set of data based on a particular characteristic or feature.

[0852] The system that realizes this invention is made up of a complex configuration including a user terminal, a server, a notification network, and an emotion engine.

[0853] Terminal

[0854] The device is assumed to be a smartphone used by the user. This smartphone is equipped with a voice recording module for recording the contents of calls. When a call is initiated, recording starts automatically and the voice data is sent to a server in real time. The device is also equipped with a microphone and a camera, which can capture the user's emotional data.

[0855] server

[0856] The server plays a central role in processing the voice and emotion data sent from the devices. The following software is installed on the server:

[0857] Generative AI model: The generative AI model uses a speech recognition module to convert the recording into text, then analyzes it for potential fraud. If fraud keywords are detected, the analysis results are scored.

[0858] Emotion Engine: The emotion engine analyzes voice tone and facial expression data to assess the user's emotional state. If the user is anxious or confused, this will also be reflected in the fraud likelihood score.

[0859] Notification Network

[0860] The server assesses the likelihood of fraud based on the results of the generative AI model and emotion engine analysis, and sends notifications to users and pre-defined trusted contacts if there is a high likelihood of fraud. The network can issue alerts via text message or in-app notification.

[0861] Specific examples

[0862] For example, if a user receives a phone call from someone who identifies themselves as a "police officer," the modules in the system will act as follows:

[0863] 1. When a user receives a call, the device's voice recording module automatically starts recording and transmits the voice data to the server in real time.

[0864] 2. A generative AI model is installed on the server, and fraud keywords such as "police" and "ID" are detected from the voice data.

[0865] 3. At the same time, the device's microphone and camera capture the user's voice tone and facial expression data, which are then analyzed by the emotion engine. It can be seen that the user's voice sounds anxious or their facial expression looks confused.

[0866] 4. Based on the analysis results of the generative AI model and the emotion engine, a high fraud score is calculated. Specifically, the generative AI model's analysis result indicates a fraud probability score of 90%, and the emotion engine's analysis result also determines that the user is in an anxious state.

[0867] 5. If the final evaluation results in a high fraud score (e.g., 90%), the server sends a notification to the user and trusted contacts stating, "A potentially fraudulent call has been detected."

[0868] An example prompt is:

[0869] "The user answers the phone and starts talking. At the same time as the voice data, facial expression data is also captured and sent to the server. Keywords such as 'police,' 'bank transfer,' and 'ID card' are detected on the server side, and at the same time the user's face shows signs of confusion or anxiety. The system rates these with a high score and notifies the user and their trusted contacts that 'A call with a high probability of fraud has been detected.'"

[0870] This allows users to be quickly warned of potential fraud and prevent it from happening. The system utilizes cutting-edge artificial intelligence and emotion recognition technology to provide an effective means of protecting users' safety in real time.

[0871] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0872] Step 1:

[0873] Start recording a call

[0874] When a user starts a call on their smartphone, the voice recording module is automatically started. The recorded voice data is temporarily stored in a buffer. In this step, the input is the call start trigger and the output is the recorded voice data.

[0875] Step 2:

[0876] Acquiring voice and emotion data

[0877] The microphone and camera on the device capture the user's voice tone and facial expression data. These data are sent to the server in real time in parallel. The input is voice and facial expression capture, and the output is voice data and emotional data.

[0878] Step 3:

[0879] Analysis of audio data

[0880] The server analyzes the received voice data using a generative AI model. The voice data is first converted into text, and fraud keywords are detected from the text data. The input is the voice data, and the output is the analyzed text data and fraud keywords.

[0881] Step 4:

[0882] Emotional Data Analysis

[0883] The server analyzes the received emotional data using an emotion engine. It evaluates the user's emotional state based on the voice tone and facial expression data. In this step, the input is the emotion data, and the output is the analyzed emotional state.

[0884] Step 5:

[0885] Fraud likelihood scoring

[0886] The server scores the likelihood of fraud based on the analysis results of the generative AI model and the analysis results of the emotional data by the emotion engine. The fraud keyword detection results are combined with the user's emotional state to calculate an overall score. The inputs are text data and emotional state, and the output is a fraud likelihood score.

[0887] Step 6:

[0888] Sending notifications

[0889] If the fraud likelihood score is high, the server will send a notification to the user and pre-defined trusted contacts. The notification will include a notification that a fraud likelihood call was detected and a warning based on the analysis results. The input is the fraud likelihood score and the output is the warning notification.

[0890] Step 7:

[0891] Call content and emotion data storage and patterning

[0892] The content and emotion data of calls that are judged to be highly likely to be fraudulent are anonymized and stored. This creates a database for future fraud prevention. The inputs are the call data and emotion data that are judged to be fraudulent, and the output is an updated fraud prevention pattern database.

[0893] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0894] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0895] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0896] [Third embodiment]

[0897] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0898] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0899] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0900] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0901] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0902] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0903] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0904] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0905] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0906] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0907] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0908] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0909] This invention is a system that records phone conversations, analyzes the recorded conversations using a generative AI model, determines the possibility of special fraud, and notifies the user. The following describes in detail an embodiment of this system.

[0910] System configuration

[0911] This system consists of the user's smartphone (terminal), a server that stores and analyzes voice data, and a network for sending notifications.

[0912] Terminal

[0913] The terminal has the function of recording voice data during a call and transmitting the data to the server in real time. When a call starts, the voice recording module is automatically started and the voice data is buffered.

[0914] server

[0915] The server receives the voice data sent from the device and processes it in real time. The generative AI model used here analyzes the voice data based on pre-trained fraud characteristics. Based on the analysis results, it scores the likelihood of fraud, and if the score exceeds a pre-set threshold, it is determined that there is a high probability of fraud. The server anonymizes and saves the content of calls that may be fraudulent, recording them as patterns.

[0916] Notification Network

[0917] The notification network sends notifications to users and pre-defined trusted contacts (e.g., family members) when potential fraud is detected via text message, email, or dedicated in-app notifications.

[0918] Program processing

[0919] Below we explain how each component of the system works.

[0920] Start recording a call (device)

[0921] When a user initiates a call on their smartphone, the device automatically records the call, storing the recording in a temporary buffer and sending it to a server in real time.

[0922] Receiving and analyzing voice data (server)

[0923] The server receives the voice data sent from the device and begins analysis using the generative AI model. The generative AI model converts the recorded data into text using a speech recognition module, and analyzes the text data to determine whether it contains any signs of fraud. As a result of the analysis, a score is assigned to the likelihood of fraud, and if the score is high, it is determined to be fraud.

[0924] Possible fraud notification (server)

[0925] If a call is deemed likely to be fraudulent, the server will send a notification to a pre-defined contact (either the user or one of their trusted contacts) stating that a potentially fraudulent call has been detected.

[0926] Save and patternize call content (server)

[0927] If a call is deemed potentially fraudulent, it is anonymized and stored in a pattern database that records the characteristics of fraud. This data will then be used to develop fraud prevention education materials and improve future AI models. Users and their families will receive regular updates on the latest fraud scams and prevention strategies.

[0928] Specific examples

[0929] System Operation Scenarios

[0930] For example, if user A receives a call from a self-proclaimed "bank representative" while on a call on his smartphone, the system operates as follows.

[0931] 1. When user A answers the phone, the device automatically records the call and sends the recording data to the server in real time.

[0932] 2. The server receives the audio data and the generative AI model analyzes it to ensure that the call contains typical scam keywords such as "bank," "account number," and "transfer."

[0933] 3. The generative AI model produces a high fraud score (e.g., 93%). The server determines that fraud is likely and prepares to send a notification.

[0934] 4. The server notifies User A and his / her family via text message that a potentially fraudulent call has been detected.

[0935] 5. The call content is anonymized and stored in a fraud pattern database.

[0936] 6. User A and his family will receive a report at a later date containing updated information on fraud prevention.

[0937] This system will quickly protect elderly people from fraud and enable them to take appropriate crime prevention measures, thereby providing an environment in which they can live their daily lives with peace of mind.

[0938] The processing flow will be explained below.

[0939] Step 1: Start recording the call (on your device)

[0940] When a user starts a call on the smartphone, the terminal detects a call start event.

[0941] The device will automatically start the audio recording module.

[0942] The device stores the recorded data in a temporary buffer and prepares it for transmission to the server in real time.

[0943] Step 2: Receiving audio data (server)

[0944] The server receives the voice data packets sent from the terminal.

[0945] The received audio data is stored in temporary storage.

[0946] The server queues the audio data for analysis by the generative AI model.

[0947] Step 3: Analyzing the voice data (server)

[0948] The server retrieves the audio data from the analysis queue.

[0949] The server uses voice recognition technology to convert the voice data into text data.

[0950] The server inputs the converted text data into the generative AI model.

[0951] The generative AI model analyzes the fraud features and calculates a fraud score.

[0952] Step 4: Determine the possibility of fraud (server)

[0953] The server evaluates the fraud scores returned by the generative AI model.

[0954] Determine whether the score exceeds a set threshold.

[0955] If fraud is deemed likely, prepare to notify.

[0956] Step 5: Sending Notifications (Server)

[0957] The server generates a notification message if fraud is deemed likely.

[0958] Notifications will be sent to the user and pre-registered contacts (family members, etc.).

[0959] Notifications will be sent via text message and in-app notifications.

[0960] Step 6: Sharing call content and patterning (server)

[0961] The server performs a process to anonymize the content of calls that are determined to be highly likely to be fraudulent.

[0962] Anonymized call transcripts are stored in a fraud pattern database.

[0963] The latest fraud methods are analyzed based on the collected data.

[0964] Regularly provide fraud prevention educational information to you and your family.

[0965] Example 1

[0966] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0967] In recent years, the methods used by special fraudsters targeting the elderly have become more sophisticated, resulting in an increase in the number of victims. To deal with such situations, it is necessary to analyze call content in real time and quickly identify dangerous calls. However, with conventional methods, recording and analyzing call content is time-consuming and laborious, making it difficult to respond quickly. In addition, there is a lack of systems that can effectively identify possible fraud and notify users in a reliable manner. To solve these issues, it is necessary to provide a system that can perform an integrated process from recording call content to detecting fraud, notifying users, and storing data.

[0968] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0969] In this invention, the server includes means for automatically recording the contents of a call when the user initiates a call, means for transmitting the recorded voice data to the server in real time, means for converting the received voice data into text data using voice recognition technology, means for analyzing the converted text data using a generative AI model to determine the possibility of fraud, means for sending a notification to the user and a designated third party if there is a possibility of fraud, and means for anonymizing and saving the contents of a call that is likely to be fraudulent and recording it in a fraud pattern database, thereby enabling real-time analysis of the contents of the call and rapid detection of fraud.

[0970] The "means for automatically recording telephone conversations" refers to a device or software that has the function of automatically recording telephone conversations when a user initiates a call and temporarily storing the recorded conversations in a buffer.

[0971] "Means for transmitting recorded voice data to a server in real time" refers to a device or software that has the function of transmitting recorded voice data to a server in real time at regular intervals or for each data size.

[0972] "Means for converting received voice data into text data using voice recognition technology" refers to a device or software that includes a voice recognition module or API for analyzing received voice data and converting its contents into text.

[0973] A "means for analyzing text data converted using a generative AI model to determine the likelihood of fraud" is a device or software that includes algorithms or methods for analyzing text data using generative AI to identify characteristics of fraud.

[0974] "Means for sending notifications to the user and designated third parties in the event of possible fraud" refers to a device or software that has the functionality to send notifications to pre-defined contacts in the event that it is determined that there is a high possibility of fraud.

[0975] "Means for anonymizing and storing phone call content that is likely to be fraudulent and recording it in a fraud pattern database" refers to a device or software that has the function of anonymizing phone call content that is determined to be likely to be fraudulent and storing the characteristics of such phone call in a database.

[0976] This invention is a system that records phone conversations, analyzes the recorded conversations using a generative AI model, determines the possibility of special fraud, and notifies the user. Specific embodiments for implementing this system are described in detail below.

[0977] System configuration

[0978] This system consists of a user's communication device (e.g., a smartphone), a server that stores and analyzes voice data, and a communication network for sending notifications.

[0979] communication terminal

[0980] When a user starts a call, the communication device automatically records the call. The recorded voice data is sent to the server in real time. The device is equipped with a voice recognition module, which automatically buffers the voice data when a call starts.

[0981] server

[0982] The server receives and processes the voice data sent from the device. The server is equipped with speech recognition technology using the Google Cloud Speech-to-Text API, which converts the voice data into text data. The converted text data is analyzed by OpenAI's generative AI model (e.g., the GPT series). This generative AI model detects keywords and phrases characteristic of fraud and scores the likelihood of fraud. If the score exceeds a set threshold, the call is deemed to have a high probability of fraud. The content of calls determined to be fraudulent is anonymized and stored in a fraud pattern database.

[0983] Notification Network

[0984] The notification network will notify users and pre-defined trusted contacts (e.g., family members) about potential fraud via text message, email, or dedicated in-app notifications.

[0985] Specific examples

[0986] System Operation Scenarios

[0987] For example, if user A receives a call from someone claiming to be a "bank representative" while on a call on his smartphone, the system operates as follows:

[0988] 1. When user A answers the phone, the device automatically records the call and sends the recording data to the server in real time.

[0989] 2. The server receives the audio data and converts it into text using the Google Cloud Speech-to-Text API.

[0990] 3. The text data is analyzed using OpenAI's generative AI model (e.g., the GPT series). For example, it checks whether the call content contains keywords such as "bank," "account number," or "transfer."

[0991] 4. The generative AI model generates a high fraud score based on the analysis results. For example, if the fraud score is evaluated as 93%, the server determines that there is a high probability of fraud and prepares to send a notification.

[0992] 5. The server notifies User A and his / her family via text message that a potentially fraudulent call has been detected.

[0993] 6. The call content is anonymized and stored in a fraud pattern database.

[0994] 7. User A and his family will receive a report at a later date containing updated information on fraud prevention.

[0995] Example prompt for a generative AI model:

[0996] Analyze calls and score them for potential fraud.

[0997] Caller details: "Hello, I'm a bank representative. What's your account number?"

[0998] This system will quickly protect elderly people from fraud and enable them to take appropriate crime prevention measures, thereby providing an environment in which they can live their daily lives with peace of mind.

[0999] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1000] Step 1: Start recording the call (on your device)

[1001] When a user starts a call on their smartphone, the device's recording module automatically starts and records the call. The recording data is temporarily stored in a buffer.

[1002] Input: Call start signal

[1003] Data processing: The recording module buffers the call audio

[1004] Output: Recorded audio data

[1005] Specific behavior: When the user presses the call button in the phone app, the recording module starts in the background and begins recording the call audio.

[1006] Step 2: Sending audio data (terminal)

[1007] The recorded audio data is sent to the server in real time using a secure encryption protocol (e.g., TLS).

[1008] Input: Buffered audio data

[1009] Data processing: Transmitted to the server in real time (encrypted)

[1010] Output: Audio data sent to the server

[1011] Specific operation: Every time the recording data reaches a certain size (e.g., 5 seconds), the device sends this data to the server.

[1012] Step 3: Receiving audio data (server)

[1013] The server receives the voice data sent from the terminal and adds it to the processing queue. The received data is encrypted to ensure security.

[1014] Input: Audio data sent from the device

[1015] Data processing: Decode received data and add it to the processing queue

[1016] Output: Audio data added to the processing queue

[1017] Specific operation: The server analyzes the arriving data packet, decodes it, and then queues it for processing.

[1018] Step 4: Converting audio data to text (server)

[1019] The server uses voice recognition technology to convert the received voice data into text data. Google Cloud Speech-to-Text API is used here.

[1020] Input: Audio data added to the processing queue

[1021] Data processing: Speech recognition and text conversion with Google Cloud Speech-to-Text API

[1022] Output: Text data

[1023] Specific operation: The server sends audio data to the API and receives the returned text data.

[1024] Step 5: Analyzing the text data (server)

[1025] The server uses a generative AI model (e.g., OpenAI's GPT) to analyze the text data, using prompts to detect keywords and phrases characteristic of fraud.

[1026] Input: Text data

[1027] Data Processing: Text Analysis with Generative AI Models

[1028] Output: Fraud likelihood score

[1029] Example prompt sentence:

[1030] Analyze the following calls and rate them for potential fraud.

[1031] Call content: {text data goes here}

[1032] Specific operation: Insert text data into the prompt sentence and call the generative AI model to perform analysis.

[1033] Step 6: Scoring Fraud Potential (Server)

[1034] Based on the results of the generative AI model's analysis, a score is assigned to the likelihood of fraud. If the score exceeds a set threshold, it is deemed highly likely to be fraudulent.

[1035] Input: Analysis results of the generative AI model

[1036] Data processing: Calculating fraud likelihood scores

[1037] Output: Fraud probability determination result (True / False)

[1038] Specific operation: The generative AI model returns a score for the analysis results and checks whether the score exceeds a set threshold.

[1039] Step 7: Sending Notifications (Server)

[1040] If fraud is deemed likely, the server will notify the user and pre-defined third parties via text message, email, or dedicated in-app notification.

[1041] Input: Fraud possibility determination result

[1042] Data processing: generating and sending notification messages

[1043] Output: Users and third parties who received the notification

[1044] What happens: The server sends an alert to the user and their contacts via a pre-registered notification channel.

[1045] Step 8: Storing and Patterning Data (Server)

[1046] Calls that are deemed likely to be fraudulent are anonymized and stored in a fraud pattern database, which is used to improve future models and for fraud prevention education.

[1047] Input: Call content determined to be fraudulent

[1048] Data processing: anonymizing call content and storing it as patterned data

[1049] Output: Data stored in the fraud pattern database

[1050] Specific actions: Calls are anonymized and stored in a fraud pattern database, which is then periodically used to improve AI models and as training materials.

[1051] (Application example 1)

[1052] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1053] As telephone fraud techniques become more sophisticated, the risk of elderly people and general users becoming victims of fraud increases. To prevent such fraud, a system is needed that can analyze call content in real time, quickly detect potentially fraudulent calls, and notify users and their trusted contacts. However, existing systems lack the accuracy of call analysis and the speed of notification, making it difficult to completely prevent fraud.

[1054] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1055] In this invention, the server includes a means for recording the contents of the call, a means for analyzing the recorded contents of the call using a generative AI model, a means for determining the possibility of special fraud based on the analysis results, a means for sending a notification when there is a possibility of special fraud, a means for sending a notification to the user and trusted contacts when the analysis results exceed a certain threshold, and a means for anonymizing and saving the contents of the call for later use as learning data or educational material for fraud prevention. This enables early detection and prompt notification of call fraud, as well as data accumulation and learning for fraud prevention.

[1056] "Call content" refers to all audio information generated during a call.

[1057] "Means of recording" refers to the technology that captures the contents of a call in digital format and stores it as data.

[1058] A "generative AI model" refers to a program that analyzes data based on patterns and features that have been previously learned using machine learning algorithms.

[1059] "Means of analysis" refers to the ability to analyze the characteristics and patterns of input data using a generative AI model.

[1060] "Special fraud" refers to a method of defrauding unspecified people of money using the telephone or the Internet.

[1061] "Means for determining" refers to the function of evaluating whether or not a specific condition is met based on the analyzed results.

[1062] "Means for sending notifications" refers to communication technologies that allow the system to notify the user or designated contacts of specific information.

[1063] "Analysis results" refers to the information and evaluations obtained as a result of data analysis by a generative AI model.

[1064] A "certain threshold" refers to a reference value set based on the analysis results, and if this reference value is exceeded, a specific action will be taken.

[1065] "User" refers to the entity that uses the system, specifically, the individual or company that makes calls using the system.

[1066] "Trusted Contacts" refers to people or entities pre-defined by the user who are authorized to receive notifications regarding fraud and other emergencies.

[1067] "Anonymization" refers to techniques that remove or change personally identifiable information to protect the privacy of data.

[1068] "Storage" refers to techniques for storing data long-term and making it available for later reuse.

[1069] "Training data" refers to past data used as training material to improve the accuracy of generative AI models.

[1070] "Educational Materials" means written, audio, video, or other educational materials used to teach fraud prevention knowledge and techniques.

[1071] This invention is a system that records phone conversations, analyzes the recorded conversations using a generative AI model, determines the possibility of special fraud, and notifies the user. Specific embodiments for implementing this system are described in detail below.

[1072] System configuration

[1073] This system consists of a user's device, a server that stores and analyzes voice data, and a network for sending notifications.

[1074] Terminal

[1075] The device has the function of recording the contents of calls and sending the recorded data to a server in real time. When a call starts, the audio recording module automatically starts and buffers the audio data. This function is realized using a recording application on the smartphone or a dedicated call recording device.

[1076] server

[1077] The server receives the voice data sent from the device and analyzes it using a generative AI model. Specifically, a voice recognition module converts the voice data into text, and then analyzes the text data to determine whether it contains any signs of fraud. This analysis could use a machine learning algorithm (such as OpenAI's GPT-4 or Google BERT). Based on the analysis results, the server scores the likelihood of fraud, and if the score exceeds a certain threshold, it determines that there is a high possibility of fraud. In addition, the content of calls determined to be fraudulent is anonymized and stored as learning data or educational material.

[1078] Notification Network

[1079] The notification network notifies users and pre-defined trusted contacts when potential fraud is detected via text message, email, or dedicated in-app notifications, using the Twilio API or Firebase Cloud Messaging.

[1080] Specific examples

[1081] When user A receives a call from a self-proclaimed "bank representative" while on a call on his smartphone, the system operates as follows.

[1082] 1. When user A answers the phone, the device automatically records the call and sends the recording data to the server in real time.

[1083] 2. The server receives the voice data and the generative AI model analyzes it to confirm that the call contains keywords characteristic of fraud, such as "bank," "account number," and "transfer."

[1084] 3. The generative AI model generates a high fraud score (e.g., 93%). The server determines that there is a high probability of fraud and sends a notification.

[1085] 4. The server notifies User A and their trusted contacts via text message that a potentially fraudulent call was detected.

[1086] 5. The call transcripts are anonymized and stored in a fraud pattern database, which will be used to improve educational materials and AI models for future fraud prevention.

[1087] Example prompt sentence:

[1088] "Please convert the audio below to text and rate it as likely to be a scam. Please also be specific about which parts are characteristic of a scam.

[1089] Audio data: [Audio data URL]

[1090] This system enables early detection and rapid notification of toll fraud, as well as data accumulation and learning for fraud prevention.

[1091] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1092] Step 1:

[1093] A user starts a call on the device. The input is "start call." When the call starts, the device automatically starts the voice recording module and starts recording the contents of the call. The voice data is temporarily stored in a buffer. The output is the recorded "voice data of the call."

[1094] Step 2:

[1095] The device sends recorded audio data to the server in real time. The input is "call audio data." The device transfers the recorded file of the call content to the server via WebSocket or REST API. The output is "sent audio data" to the server.

[1096] Step 3:

[1097] The server receives the voice data and begins analysis using a generative AI model. The input is the "received voice data." The server converts the voice data into text using a speech recognition module. The text data is then input into a generative AI model (for example, OpenAI's GPT-4) for analysis. This detects whether the voice data contains features or patterns that indicate possible fraud. The output is the "analysis results."

[1098] Step 4:

[1099] The server scores the likelihood of fraud based on the analysis results. The input is the "analysis results." The generative AI model evaluates the likelihood of fraud based on prior learning and generates a score (e.g., a fraud score of 93%). The output is the "fraud score."

[1100] Step 5:

[1101] The server determines if the fraud score exceeds a certain threshold and prepares to send a notification if so. The inputs are the "fraud score" and the "threshold". In the case of a high fraud score above the threshold, the server generates a notification message and prepares to send it. The output is the "notification message".

[1102] Step 6:

[1103] The server sends notifications to the user and trusted contacts. The inputs are a "notification message" and "contact information." The server uses the Twilio API or Firebase Cloud Messaging to send notifications to inform the user and trusted contacts of possible fraud. The output is a "sent notification."

[1104] Step 7:

[1105] The server anonymizes the content of calls determined to be fraudulent and stores it in a pattern database. The input is "content of calls determined to be fraudulent." The server anonymizes the content of calls and stores it in a fraud pattern database for use as future learning data or educational material. The output is "stored anonymized data."

[1106] Step 8:

[1107] The server generates and distributes periodic reports to users and their families so that they can receive the latest information on fraud prevention. The inputs are "stored anonymized data" and "report distribution schedule." The server generates reports summarizing the latest fraud techniques and prevention strategies based on the stored data and distributes them periodically to users and trusted contacts. The output is "sent reports."

[1108] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1109] This invention is a system that records phone conversations, analyzes the recordings using a generative AI model and an emotion engine, determines the possibility of special fraud, and notifies the user if necessary. The following describes in detail an embodiment of this system.

[1110] System configuration

[1111] This system consists of the user's smartphone (device), a server that stores and analyzes voice data, a network for sending notifications, and an emotion engine that recognizes the user's emotions.

[1112] Terminal

[1113] The device has the function of recording the user's voice during a call and sending the data to the server in real time. When a call starts, the voice recording module automatically starts and buffers the voice data. The device is also equipped with a microphone and camera to capture the user's emotional data.

[1114] server

[1115] The server receives the voice and emotion data sent from the device and processes it in real time. The voice data is analyzed by a generative AI model to calculate a fraud probability score. The emotion engine also analyzes the user's emotional state.

[1116] Emotion Engine

[1117] The emotion engine is a module that recognizes the user's emotions from voice and image data. The emotion engine analyzes changes in voice tone and facial expressions to identify emotions such as nervousness, confusion, or surprise.

[1118] Notification Network

[1119] The notification network sends notifications to users and pre-defined trusted contacts (e.g., family members) when fraud is deemed likely, and can take emotional data into account to provide even more accurate warnings.

[1120] Program processing

[1121] Below we explain how each component of the system works.

[1122] Start recording a call (device)

[1123] When a user initiates a call on their smartphone, the device automatically records the call, storing the recording in a temporary buffer and sending it to a server in real time.

[1124] Receiving and analyzing voice data (server)

[1125] The server receives the voice data sent from the device and begins analysis using the generative AI model. The generative AI model converts the recorded data into text using a speech recognition module, and analyzes the text data to determine whether it contains any signs of fraud. As a result of the analysis, a score is assigned to the likelihood of fraud, and a final evaluation is made along with the user's emotional data using an emotion engine.

[1126] Acquisition and analysis of emotion data (device and server)

[1127] The device's microphone and camera capture the user's voice and facial expression data, which is then sent to a server that analyzes the data using an emotion engine to assess the user's emotional state. If the user expresses anxiety or confusion, this is also factored into the fraud likelihood score.

[1128] Possible fraud notification (server)

[1129] If a call is deemed likely to be fraudulent, the server sends a notification to pre-defined contacts (the user or one of their trusted contacts) informing them that a potentially fraudulent call has been detected and providing a warning based on the user's emotional state.

[1130] Storage and patterning of call content and emotion data (server)

[1131] If the call content and emotional data is determined to be potentially fraudulent, it is anonymized and stored in a database that records the characteristics of the fraud and the user's emotional state. This data will then be used to develop fraud prevention education materials and improve future AI models. Users and their families will receive regular updates on the latest fraud scams and prevention strategies.

[1132] Specific examples

[1133] System Operation Scenarios

[1134] For example, if user B receives a call from a self-proclaimed "police officer" while on a call on his smartphone, the system operates as follows.

[1135] 1. When User B answers the phone, the device automatically records the call and simultaneously captures emotion data.

[1136] 2. The recorded data and emotional data are sent to the server in real time.

[1137] 3. The generative AI model analyzes the voice data and detects fraud keywords such as "police" and "ID." In addition, the emotion engine detects User B's anxious voice and confused facial expression.

[1138] 4. Based on a high fraud score (e.g., 90%) and User B's emotional state, fraud is deemed more likely.

[1139] 5. The server sends a text message to User B and his family saying, "A likely fraudulent call has been detected. User B is in a state of anxiety."

[1140] 6. The call content and emotional data is anonymized and stored in a fraud pattern database.

[1141] 7. User B and his family will receive a report at a later date containing updated information on fraud prevention.

[1142] This system will enable more accurate fraud detection based on the user's emotional state, providing quick and accurate protection from fraud. As a result, it will provide an environment where all users, including the elderly, can make calls with peace of mind.

[1143] The processing flow will be explained below.

[1144] Step 1: Start recording the call (on your device)

[1145] When a user starts a call on the smartphone, the terminal detects a call start event.

[1146] The device will automatically start the audio recording module.

[1147] The device stores the recorded data in a temporary buffer and prepares it for transmission to the server in real time.

[1148] Step 2: Receiving audio data (server)

[1149] The server receives the voice data packets sent from the terminal.

[1150] The received audio data is stored in a temporary storage device.

[1151] The server queues the audio data for analysis by the generative AI model.

[1152] Step 3: Analyzing the voice data (server)

[1153] The server retrieves the audio data from the analysis queue.

[1154] The server uses voice recognition technology to convert the voice data into text data.

[1155] The server uses a generative AI model to analyze the text data and detect fraudulent features.

[1156] Based on the analysis results, a fraud likelihood score is calculated.

[1157] Step 4: Acquiring emotion data (device)

[1158] The device's microphone and camera capture the user's voice and facial expression data.

[1159] The terminal transmits audio and image data to the server in real time.

[1160] Step 5: Emotion data analysis (server)

[1161] The server uses an emotion engine to analyze the received audio and image data.

[1162] An emotion engine identifies the user's emotional state (e.g., anxiety, confusion, surprise, etc.).

[1163] The analysis results of the emotion data and voice data are combined to update the final fraud likelihood score.

[1164] Step 6: Determine the likelihood of fraud (server)

[1165] The server evaluates the updated fraud likelihood score.

[1166] Determine whether the score exceeds a set threshold.

[1167] If fraud is deemed likely, prepare to notify.

[1168] Step 7: Sending Notifications (Server)

[1169] The server generates a notification message if fraud is deemed likely.

[1170] The notification includes information about the likelihood of fraud and the user's emotional state.

[1171] Notifications will be sent to the user and pre-registered contacts (such as family members) via text message or in-app notification.

[1172] Step 8: Storing and patterning call content and emotion data (server)

[1173] The server anonymizes the content of calls that are deemed likely to be fraudulent.

[1174] The anonymized data, including emotional data, is stored in a fraud pattern database.

[1175] The latest fraud methods are analyzed based on the collected data.

[1176] Regularly provide fraud prevention educational information to you and your family.

[1177] Example 2

[1178] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1179] Conventional call recording systems have low detection accuracy for special frauds, making it difficult to prevent fraud victims. Furthermore, because they do not take into account the user's emotional state, they are unable to properly assess the anxiety and confusion felt by the user. Furthermore, when a potential fraud is detected, notification methods are limited, making it impossible to quickly communicate warnings to a wide range of contacts. These issues need to be resolved.

[1180] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1181] In this invention, the server includes a means for recording the content of the call, a means for analyzing the recorded content of the call using a generative AI model, a means for determining the possibility of special fraud based on the analysis results, a means for acquiring and analyzing emotional data, and a means for sending a notification if there is a possibility of special fraud. This allows for simultaneous analysis of the content of the call and the user's emotional state, enabling more accurate fraud detection and faster notification.

[1182] A "call recording device" is a device or software that allows a user to capture and save audio data in real time while making a call.

[1183] "Means of analysis using a generative AI model" refers to artificial intelligence technology that converts recorded audio data into text data and analyzes the characteristics of fraud.

[1184] "Means for determining the possibility of special fraud" refers to algorithms or programs for evaluating and scoring the possibility of fraud based on the analysis results of the generative AI model.

[1185] "Means for acquiring and analyzing emotional data" refers to emotion recognition technology for capturing a user's vocal tone and facial expressions and analyzing the data to assess the user's emotional state.

[1186] "Means for sending notifications" refers to communication means for sending information to the user and pre-defined trusted contacts if fraud is determined to be likely.

[1187] This invention is a system that records phone conversations, analyzes the recordings using a generative AI model and an emotion engine, determines the possibility of special fraud, and notifies the user if necessary. The following describes in detail an embodiment of this system.

[1188] System configuration

[1189] This system consists of the user's smartphone (device), a server that stores and analyzes voice data, a network for sending notifications, and an emotion engine that recognizes the user's emotions.

[1190] Terminal

[1191] The device has the function of recording the user's voice during a call and sending the data to the server in real time. When a call starts, the voice recording module automatically starts and buffers the voice data. The device is also equipped with a microphone and camera to capture the user's emotional data.

[1192] server

[1193] The server receives the voice and emotion data sent from the device and processes it in real time. The voice data is analyzed by a generative AI model to calculate a fraud probability score. The emotion engine also analyzes the user's emotional state.

[1194] Emotion Engine

[1195] The emotion engine is a module that recognizes the user's emotions from voice and image data. The emotion engine analyzes changes in voice tone and facial expressions to identify emotions such as nervousness, confusion, or surprise.

[1196] Notification Network

[1197] The notification network sends notifications to users and pre-defined trusted contacts (e.g., family members) when fraud is deemed likely, and can take emotional data into account to provide even more accurate warnings.

[1198] Program processing

[1199] Below we explain how each component of the system works.

[1200] Start recording a call (device)

[1201] When a user starts a call on their smartphone, the device automatically records the conversation. This recording is temporarily stored in a buffer and then sent to a server in real time. The device is also equipped with a microphone and camera to capture the user's emotional data.

[1202] Receiving and analyzing voice data (server)

[1203] The server receives the voice data sent from the device and begins analysis using the generative AI model. Specifically, it uses a voice recognition module to convert the recorded data into text data, and then analyzes the text data to determine whether it contains any fraudulent characteristics. This analysis calculates a fraud likelihood score. The emotion engine simultaneously analyzes the user's emotional data, and a final evaluation is made.

[1204] Acquisition and analysis of emotion data (device and server)

[1205] The device's microphone and camera capture the user's voice and facial expression data, which is then sent to a server that analyzes it using an emotion engine to assess the user's emotional state. If the user expresses anxiety or confusion, this is also factored into the fraud likelihood score.

[1206] Possible fraud notification (server)

[1207] If a fraudulent call is determined to be likely, the server sends a notification to a pre-defined contact with the message, "A fraudulent call has been detected. User B is in a state of anxiety."

[1208] Storage and patterning of call content and emotion data (server)

[1209] If a call is deemed potentially fraudulent, the call's content and emotional data will be stored in an anonymized database. This data will then be used to improve fraud prevention education materials and AI models. Users and their families will also receive regular updates on the latest fraud scams and prevention strategies.

[1210] Example: System Operation Scenarios

[1211] For example, if a user receives a call from a self-proclaimed "police officer" while on a call on their smartphone, the system operates as follows.

[1212] When the user answers the call, the device automatically records the call and simultaneously captures emotional data.

[1213] Recorded data and emotional data are sent to the server in real time.

[1214] A generative AI model analyzes the audio data to detect fraud keywords such as "police" and "ID," while an emotion engine detects anxious voices and confused facial expressions.

[1215] A high fraud score (e.g., 90%) and the user's emotional state are evaluated to determine the likelihood of fraud.

[1216] The server sends a text message to the user and their family saying, "A likely fraudulent call has been detected. The user is in a state of anxiety."

[1217] The call content and emotional data is anonymized and stored in a fraud pattern database.

[1218] Users and their families will receive a report at a later date containing updated information on fraud prevention.

[1219] Prompt Sentence Examples

[1220] "Specify how a system works to detect potential fraud while a user is on a call."

[1221] This system will enable more accurate fraud detection based on the user's emotional state, providing quick and accurate protection from fraud. As a result, it will provide an environment where all users, including the elderly, can make calls with peace of mind.

[1222] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1223] Step 1:

[1224] When a user initiates a call on their smartphone, the device automatically records the call, temporarily stores the recording in a buffer, and then transmits it to a server in real time.

[1225] Specific behavior:

[1226] 1. When a user makes or receives a call, the smartphone's voice recording module is automatically activated.

[1227] 2. Audio data is captured through the built-in microphone and temporarily stored in the device's memory.

[1228] 3. The buffered data is encrypted and sent to the server in real time.

[1229] Input: Call audio data

[1230] Output: Encrypted recording data

[1231] Step 2:

[1232] The server receives the voice data sent from the device and temporarily stores the recorded data in a database, while simultaneously starting analysis using the generative AI model.

[1233] Specific behavior:

[1234] 1. The server receives the encrypted voice data, decrypts it, and temporarily stores it in a database.

[1235] 2. The decoded voice data is input into the generative AI model and converted into text data using a voice recognition module.

[1236] 3. A generative AI model analyzes the text data and calculates a fraud likelihood score.

[1237] Input: Encrypted audio recording

[1238] Output: Analysis results including fraud likelihood score

[1239] Step 3:

[1240] The device's microphone and camera capture the user's voice and facial expression data, which is then sent to a server.

[1241] Specific behavior:

[1242] 1. The device captures the user's voice tone with the microphone.

[1243] 2. The camera captures the user's facial expressions in real time.

[1244] 3. The acquired audio and image data are sent to the server.

[1245] Input: Voice tone and facial expression data

[1246] Output: Voice tone and facial expression data sent to the server

[1247] Step 4:

[1248] The server receives the emotion data sent from the terminal and analyzes it using an emotion engine.

[1249] Specific behavior:

[1250] 1. The server inputs the received voice tone and facial expression data into the emotion engine.

[1251] 2. The emotion engine analyzes voice tone and facial expressions to assess the user's emotional state.

[1252] 3. The emotional state assessment results are combined with the deception likelihood score to form a final assessment.

[1253] Input: Voice tone and facial expression data

[1254] Output: Evaluation results including the user's emotional state

[1255] Step 5:

[1256] If fraud is deemed likely, the server will send a notification to pre-defined contacts.

[1257] Specific behavior:

[1258] 1. The server makes a comprehensive assessment based on the fraud probability score and the user's emotional state.

[1259] 2. Generate a notification message if fraud is determined to be likely.

[1260] 3. Send a notification message to you and your pre-defined trusted contacts.

[1261] Input: Evaluation results including the user's emotional state

[1262] Output: Notification message

[1263] Step 6:

[1264] The server anonymizes the content of calls that are deemed likely to be fraudulent and the user's emotional data and stores them in a database.

[1265] Specific behavior:

[1266] 1. Anonymize data that the analysis assesses as likely to be fraudulent.

[1267] 2. Store the anonymized data in a fraud pattern database.

[1268] 3. The data will later be used to improve fraud prevention educational materials and AI models.

[1269] Input: Call content and emotion data including evaluation results

[1270] Output: Anonymized database records

[1271] (Application example 2)

[1272] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1273] In today's society, the number of victims of telephone fraud is increasing, and elderly people in particular are prone to falling prey to the sophisticated tactics of fraudsters. Furthermore, conventional fraud prevention systems have issues with their difficulty in accurately identifying fraud and their inability to issue warnings at the appropriate time because they do not take the user's emotional state into account. This invention aims to provide a system that can accurately detect the possibility of fraud and issue warnings at the appropriate time by analyzing the content of phone calls and comprehensively assessing the user's emotional state.

[1274] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1275] In this invention, the server includes a means for recording call content, a means for analyzing the recorded call content with a generative AI model, a means for detecting fraud keywords, a means for acquiring and analyzing user emotion data, a means for scoring the likelihood of fraud based on the analysis results and the emotion data, and a means for sending a notification to the user and trusted contacts if the likelihood of fraud is high, thereby enabling advanced fraud detection and warning based not only on call content but also on the user's emotional state.

[1276] "Call content" refers to the audio and text data exchanged during a phone or video call.

[1277] "Recording" is the process of storing telephone conversations in digital or analog form.

[1278] A "generative AI model" is an artificial intelligence model that uses machine learning algorithms to analyze voice data and identify specific patterns and keywords.

[1279] "Analysis" is the process of evaluating recorded data to detect specific patterns or characteristics.

[1280] "Fraud keywords" are specific words or phrases used to characterize fraudulent activity.

[1281] "Emotion data" refers to data that includes changes in voice tone and facial expressions that indicate the emotional state of the user during a call.

[1282] "Scoring" is the process of evaluating data based on specific criteria and assigning them a score accordingly.

[1283] "Notification" is a communication method used to convey alerts or information to users or trusted contacts.

[1284] "Trusted Contacts" refers to people or organizations that a user pre-selects with whom they can share emergency or important information.

[1285] "Anonymization" is the process of removing personally identifiable information from data so that it cannot be identified by third parties.

[1286] A "pattern" refers to an arrangement or tendency of a set of data based on a particular characteristic or feature.

[1287] The system that realizes this invention is made up of a complex configuration including a user terminal, a server, a notification network, and an emotion engine.

[1288] Terminal

[1289] The device is assumed to be a smartphone used by the user. This smartphone is equipped with a voice recording module for recording the contents of calls. When a call is initiated, recording starts automatically and the voice data is sent to a server in real time. The device is also equipped with a microphone and a camera, which can capture the user's emotional data.

[1290] server

[1291] The server plays a central role in processing the voice and emotion data sent from the devices. The following software is installed on the server:

[1292] Generative AI model: The generative AI model uses a speech recognition module to convert the recording into text, then analyzes it for potential fraud. If fraud keywords are detected, the analysis results are scored.

[1293] Emotion Engine: The emotion engine analyzes voice tone and facial expression data to assess the user's emotional state. If the user is anxious or confused, this will also be reflected in the fraud likelihood score.

[1294] Notification Network

[1295] The server assesses the likelihood of fraud based on the results of the generative AI model and emotion engine analysis, and sends notifications to users and pre-defined trusted contacts if there is a high likelihood of fraud. The network can issue alerts via text message or in-app notification.

[1296] Specific examples

[1297] For example, if a user receives a phone call from someone who identifies themselves as a "police officer," the modules in the system will act as follows:

[1298] 1. When a user receives a call, the device's voice recording module automatically starts recording and transmits the voice data to the server in real time.

[1299] 2. A generative AI model is installed on the server, and fraud keywords such as "police" and "ID" are detected from the voice data.

[1300] 3. At the same time, the device's microphone and camera capture the user's voice tone and facial expression data, which are then analyzed by the emotion engine. It can be seen that the user's voice sounds anxious or their facial expression looks confused.

[1301] 4. Based on the analysis results of the generative AI model and the emotion engine, a high fraud score is calculated. Specifically, the generative AI model's analysis result indicates a fraud probability score of 90%, and the emotion engine's analysis result also determines that the user is in an anxious state.

[1302] 5. If the final evaluation results in a high fraud score (e.g., 90%), the server sends a notification to the user and trusted contacts stating, "A potentially fraudulent call has been detected."

[1303] An example prompt is:

[1304] "The user answers the phone and starts talking. At the same time as the voice data, facial expression data is also captured and sent to the server. Keywords such as 'police,' 'bank transfer,' and 'ID card' are detected on the server side, and at the same time the user's face shows signs of confusion or anxiety. The system rates these with a high score and notifies the user and their trusted contacts that 'A call with a high probability of fraud has been detected.'"

[1305] This allows users to be quickly warned of potential fraud and prevent it from happening. The system utilizes cutting-edge artificial intelligence and emotion recognition technology to provide an effective means of protecting users' safety in real time.

[1306] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1307] Step 1:

[1308] Start recording a call

[1309] When a user starts a call on their smartphone, the voice recording module is automatically started. The recorded voice data is temporarily stored in a buffer. In this step, the input is the call start trigger and the output is the recorded voice data.

[1310] Step 2:

[1311] Acquiring voice and emotion data

[1312] The microphone and camera on the device capture the user's voice tone and facial expression data. These data are sent to the server in real time in parallel. The input is voice and facial expression capture, and the output is voice data and emotional data.

[1313] Step 3:

[1314] Analysis of audio data

[1315] The server analyzes the received voice data using a generative AI model. The voice data is first converted into text, and fraud keywords are detected from the text data. The input is the voice data, and the output is the analyzed text data and fraud keywords.

[1316] Step 4:

[1317] Emotional Data Analysis

[1318] The server analyzes the received emotional data using an emotion engine. It evaluates the user's emotional state based on the voice tone and facial expression data. In this step, the input is the emotion data, and the output is the analyzed emotional state.

[1319] Step 5:

[1320] Fraud likelihood scoring

[1321] The server scores the likelihood of fraud based on the analysis results of the generative AI model and the analysis results of the emotional data by the emotion engine. The fraud keyword detection results are combined with the user's emotional state to calculate an overall score. The inputs are text data and emotional state, and the output is a fraud likelihood score.

[1322] Step 6:

[1323] Sending notifications

[1324] If the fraud likelihood score is high, the server will send a notification to the user and pre-defined trusted contacts. The notification will include a notification that a fraud likelihood call was detected and a warning based on the analysis results. The input is the fraud likelihood score and the output is the warning notification.

[1325] Step 7:

[1326] Call content and emotion data storage and patterning

[1327] The content and emotion data of calls that are judged to be highly likely to be fraudulent are anonymized and stored. This creates a database for future fraud prevention. The inputs are the call data and emotion data that are judged to be fraudulent, and the output is an updated fraud prevention pattern database.

[1328] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1329] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1330] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1331] [Fourth embodiment]

[1332] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1333] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1334] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1335] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1336] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1337] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1338] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1339] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1340] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1341] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1342] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1343] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1344] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1345] This invention is a system that records phone conversations, analyzes the recorded conversations using a generative AI model, determines the possibility of special fraud, and notifies the user. The following describes in detail an embodiment of this system.

[1346] System configuration

[1347] This system consists of the user's smartphone (terminal), a server that stores and analyzes voice data, and a network for sending notifications.

[1348] Terminal

[1349] The terminal has the function of recording voice data during a call and transmitting the data to the server in real time. When a call starts, the voice recording module is automatically started and the voice data is buffered.

[1350] server

[1351] The server receives the voice data sent from the device and processes it in real time. The generative AI model used here analyzes the voice data based on pre-trained fraud characteristics. Based on the analysis results, it scores the likelihood of fraud, and if the score exceeds a pre-set threshold, it is determined that there is a high probability of fraud. The server anonymizes and saves the content of calls that may be fraudulent, recording them as patterns.

[1352] Notification Network

[1353] The notification network sends notifications to users and pre-defined trusted contacts (e.g., family members) when potential fraud is detected via text message, email, or dedicated in-app notifications.

[1354] Program processing

[1355] Below we explain how each component of the system works.

[1356] Start recording a call (device)

[1357] When a user initiates a call on their smartphone, the device automatically records the call, storing the recording in a temporary buffer and sending it to a server in real time.

[1358] Receiving and analyzing voice data (server)

[1359] The server receives the voice data sent from the device and begins analysis using the generative AI model. The generative AI model converts the recorded data into text using a speech recognition module, and analyzes the text data to determine whether it contains any signs of fraud. As a result of the analysis, a score is assigned to the likelihood of fraud, and if the score is high, it is determined to be fraud.

[1360] Possible fraud notification (server)

[1361] If a call is deemed likely to be fraudulent, the server will send a notification to a pre-defined contact (either the user or one of their trusted contacts) stating that a potentially fraudulent call has been detected.

[1362] Save and patternize call content (server)

[1363] If a call is deemed potentially fraudulent, it is anonymized and stored in a pattern database that records the characteristics of fraud. This data will then be used to develop fraud prevention education materials and improve future AI models. Users and their families will receive regular updates on the latest fraud scams and prevention strategies.

[1364] Specific examples

[1365] System Operation Scenarios

[1366] For example, if user A receives a call from a self-proclaimed "bank representative" while on a call on his smartphone, the system operates as follows.

[1367] 1. When user A answers the phone, the device automatically records the call and sends the recording data to the server in real time.

[1368] 2. The server receives the audio data and the generative AI model analyzes it to ensure that the call contains typical scam keywords such as "bank," "account number," and "transfer."

[1369] 3. The generative AI model produces a high fraud score (e.g., 93%). The server determines that fraud is likely and prepares to send a notification.

[1370] 4. The server notifies User A and his / her family via text message that a potentially fraudulent call has been detected.

[1371] 5. The call content is anonymized and stored in a fraud pattern database.

[1372] 6. User A and his family will receive a report at a later date containing updated information on fraud prevention.

[1373] This system will quickly protect elderly people from fraud and enable them to take appropriate crime prevention measures, thereby providing an environment in which they can live their daily lives with peace of mind.

[1374] The processing flow will be explained below.

[1375] Step 1: Start recording the call (on your device)

[1376] When a user starts a call on the smartphone, the terminal detects a call start event.

[1377] The device will automatically start the audio recording module.

[1378] The device stores the recorded data in a temporary buffer and prepares it for transmission to the server in real time.

[1379] Step 2: Receiving audio data (server)

[1380] The server receives the voice data packets sent from the terminal.

[1381] The received audio data is stored in temporary storage.

[1382] The server queues the audio data for analysis by the generative AI model.

[1383] Step 3: Analyzing the voice data (server)

[1384] The server retrieves the audio data from the analysis queue.

[1385] The server uses voice recognition technology to convert the voice data into text data.

[1386] The server inputs the converted text data into the generative AI model.

[1387] The generative AI model analyzes the fraud features and calculates a fraud score.

[1388] Step 4: Determine the possibility of fraud (server)

[1389] The server evaluates the fraud scores returned by the generative AI model.

[1390] Determine whether the score exceeds a set threshold.

[1391] If fraud is deemed likely, prepare to notify.

[1392] Step 5: Sending Notifications (Server)

[1393] The server generates a notification message if fraud is deemed likely.

[1394] Notifications will be sent to the user and pre-registered contacts (family members, etc.).

[1395] Notifications will be sent via text message and in-app notifications.

[1396] Step 6: Sharing call content and patterning (server)

[1397] The server performs a process to anonymize the content of calls that are determined to be highly likely to be fraudulent.

[1398] Anonymized call transcripts are stored in a fraud pattern database.

[1399] The latest fraud methods are analyzed based on the collected data.

[1400] Regularly provide fraud prevention educational information to you and your family.

[1401] Example 1

[1402] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1403] In recent years, the methods used by special fraudsters targeting the elderly have become more sophisticated, resulting in an increase in the number of victims. To deal with such situations, it is necessary to analyze call content in real time and quickly identify dangerous calls. However, with conventional methods, recording and analyzing call content is time-consuming and laborious, making it difficult to respond quickly. In addition, there is a lack of systems that can effectively identify possible fraud and notify users in a reliable manner. To solve these issues, it is necessary to provide a system that can perform an integrated process from recording call content to detecting fraud, notifying users, and storing data.

[1404] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1405] In this invention, the server includes means for automatically recording the contents of a call when the user initiates a call, means for transmitting the recorded voice data to the server in real time, means for converting the received voice data into text data using voice recognition technology, means for analyzing the converted text data using a generative AI model to determine the possibility of fraud, means for sending a notification to the user and a designated third party if there is a possibility of fraud, and means for anonymizing and saving the contents of a call that is likely to be fraudulent and recording it in a fraud pattern database, thereby enabling real-time analysis of the contents of the call and rapid detection of fraud.

[1406] The "means for automatically recording telephone conversations" refers to a device or software that has the function of automatically recording telephone conversations when a user initiates a call and temporarily storing the recorded conversations in a buffer.

[1407] "Means for transmitting recorded voice data to a server in real time" refers to a device or software that has the function of transmitting recorded voice data to a server in real time at regular intervals or for each data size.

[1408] "Means for converting received voice data into text data using voice recognition technology" refers to a device or software that includes a voice recognition module or API for analyzing received voice data and converting its contents into text.

[1409] A "means for analyzing text data converted using a generative AI model to determine the likelihood of fraud" is a device or software that includes algorithms or methods for analyzing text data using generative AI to identify characteristics of fraud.

[1410] "Means for sending notifications to the user and designated third parties in the event of possible fraud" refers to a device or software that has the functionality to send notifications to pre-defined contacts in the event that it is determined that there is a high possibility of fraud.

[1411] "Means for anonymizing and storing phone call content that is likely to be fraudulent and recording it in a fraud pattern database" refers to a device or software that has the function of anonymizing phone call content that is determined to be likely to be fraudulent and storing the characteristics of such phone call in a database.

[1412] This invention is a system that records phone conversations, analyzes the recorded conversations using a generative AI model, determines the possibility of special fraud, and notifies the user. Specific embodiments for implementing this system are described in detail below.

[1413] System configuration

[1414] This system consists of a user's communication device (e.g., a smartphone), a server that stores and analyzes voice data, and a communication network for sending notifications.

[1415] communication terminal

[1416] When a user starts a call, the communication device automatically records the call. The recorded voice data is sent to the server in real time. The device is equipped with a voice recognition module, which automatically buffers the voice data when a call starts.

[1417] server

[1418] The server receives and processes the voice data sent from the device. The server is equipped with speech recognition technology using the Google Cloud Speech-to-Text API, which converts the voice data into text data. The converted text data is analyzed by OpenAI's generative AI model (e.g., the GPT series). This generative AI model detects keywords and phrases characteristic of fraud and scores the likelihood of fraud. If the score exceeds a set threshold, the call is deemed to have a high probability of fraud. The content of calls determined to be fraudulent is anonymized and stored in a fraud pattern database.

[1419] Notification Network

[1420] The notification network will notify users and pre-defined trusted contacts (e.g., family members) about potential fraud via text message, email, or dedicated in-app notifications.

[1421] Specific examples

[1422] System Operation Scenarios

[1423] For example, if user A receives a call from someone claiming to be a "bank representative" while on a call on his smartphone, the system operates as follows:

[1424] 1. When user A answers the phone, the device automatically records the call and sends the recording data to the server in real time.

[1425] 2. The server receives the audio data and converts it into text using the Google Cloud Speech-to-Text API.

[1426] 3. The text data is analyzed using OpenAI's generative AI model (e.g., the GPT series). For example, it checks whether the call content contains keywords such as "bank," "account number," or "transfer."

[1427] 4. The generative AI model generates a high fraud score based on the analysis results. For example, if the fraud score is evaluated as 93%, the server determines that there is a high probability of fraud and prepares to send a notification.

[1428] 5. The server notifies User A and his / her family via text message that a potentially fraudulent call has been detected.

[1429] 6. The call content is anonymized and stored in a fraud pattern database.

[1430] 7. User A and his family will receive a report at a later date containing updated information on fraud prevention.

[1431] Example prompt for a generative AI model:

[1432] Analyze calls and score them for potential fraud.

[1433] Caller details: "Hello, I'm a bank representative. What's your account number?"

[1434] This system will quickly protect elderly people from fraud and enable them to take appropriate crime prevention measures, thereby providing an environment in which they can live their daily lives with peace of mind.

[1435] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1436] Step 1: Start recording the call (on your device)

[1437] When a user starts a call on their smartphone, the device's recording module automatically starts and records the call. The recording data is temporarily stored in a buffer.

[1438] Input: Call start signal

[1439] Data processing: The recording module buffers the call audio

[1440] Output: Recorded audio data

[1441] Specific behavior: When the user presses the call button in the phone app, the recording module starts in the background and begins recording the call audio.

[1442] Step 2: Sending audio data (terminal)

[1443] The recorded audio data is sent to the server in real time using a secure encryption protocol (e.g., TLS).

[1444] Input: Buffered audio data

[1445] Data processing: Transmitted to the server in real time (encrypted)

[1446] Output: Audio data sent to the server

[1447] Specific operation: Every time the recording data reaches a certain size (e.g., 5 seconds), the device sends this data to the server.

[1448] Step 3: Receiving audio data (server)

[1449] The server receives the voice data sent from the terminal and adds it to the processing queue. The received data is encrypted to ensure security.

[1450] Input: Audio data sent from the device

[1451] Data processing: Decode received data and add it to the processing queue

[1452] Output: Audio data added to the processing queue

[1453] Specific operation: The server analyzes the arriving data packet, decodes it, and then queues it for processing.

[1454] Step 4: Converting audio data to text (server)

[1455] The server uses voice recognition technology to convert the received voice data into text data. Google Cloud Speech-to-Text API is used here.

[1456] Input: Audio data added to the processing queue

[1457] Data processing: Speech recognition and text conversion with Google Cloud Speech-to-Text API

[1458] Output: Text data

[1459] Specific operation: The server sends audio data to the API and receives the returned text data.

[1460] Step 5: Analyzing the text data (server)

[1461] The server uses a generative AI model (e.g., OpenAI's GPT) to analyze the text data, using prompts to detect keywords and phrases characteristic of fraud.

[1462] Input: Text data

[1463] Data Processing: Text Analysis with Generative AI Models

[1464] Output: Fraud likelihood score

[1465] Example prompt sentence:

[1466] Analyze the following calls and rate them for potential fraud.

[1467] Call content: {text data goes here}

[1468] Specific operation: Insert text data into the prompt sentence and call the generative AI model to perform analysis.

[1469] Step 6: Scoring Fraud Potential (Server)

[1470] Based on the results of the generative AI model's analysis, a score is assigned to the likelihood of fraud. If the score exceeds a set threshold, it is deemed highly likely to be fraudulent.

[1471] Input: Analysis results of the generative AI model

[1472] Data processing: Calculating fraud likelihood scores

[1473] Output: Fraud probability determination result (True / False)

[1474] Specific operation: The generative AI model returns a score for the analysis results and checks whether the score exceeds a set threshold.

[1475] Step 7: Sending Notifications (Server)

[1476] If fraud is deemed likely, the server will notify the user and pre-defined third parties via text message, email, or dedicated in-app notification.

[1477] Input: Fraud possibility determination result

[1478] Data processing: generating and sending notification messages

[1479] Output: Users and third parties who received the notification

[1480] What happens: The server sends an alert to the user and their contacts via a pre-registered notification channel.

[1481] Step 8: Storing and Patterning Data (Server)

[1482] Calls that are deemed likely to be fraudulent are anonymized and stored in a fraud pattern database, which is used to improve future models and for fraud prevention education.

[1483] Input: Call content determined to be fraudulent

[1484] Data processing: anonymizing call content and storing it as patterned data

[1485] Output: Data stored in the fraud pattern database

[1486] Specific actions: Calls are anonymized and stored in a fraud pattern database, which is then periodically used to improve AI models and as training materials.

[1487] (Application example 1)

[1488] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1489] As telephone fraud techniques become more sophisticated, the risk of elderly people and general users becoming victims of fraud increases. To prevent such fraud, a system is needed that can analyze call content in real time, quickly detect potentially fraudulent calls, and notify users and their trusted contacts. However, existing systems lack the accuracy of call analysis and the speed of notification, making it difficult to completely prevent fraud.

[1490] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1491] In this invention, the server includes a means for recording the contents of the call, a means for analyzing the recorded contents of the call using a generative AI model, a means for determining the possibility of special fraud based on the analysis results, a means for sending a notification when there is a possibility of special fraud, a means for sending a notification to the user and trusted contacts when the analysis results exceed a certain threshold, and a means for anonymizing and saving the contents of the call for later use as learning data or educational material for fraud prevention. This enables early detection and prompt notification of call fraud, as well as data accumulation and learning for fraud prevention.

[1492] "Call content" refers to all audio information generated during a call.

[1493] "Means of recording" refers to the technology that captures the contents of a call in digital format and stores it as data.

[1494] A "generative AI model" refers to a program that analyzes data based on patterns and features that have been previously learned using machine learning algorithms.

[1495] "Means of analysis" refers to the ability to analyze the characteristics and patterns of input data using a generative AI model.

[1496] "Special fraud" refers to a method of defrauding unspecified people of money using the telephone or the Internet.

[1497] "Means for determining" refers to the function of evaluating whether or not a specific condition is met based on the analyzed results.

[1498] "Means for sending notifications" refers to communication technologies that allow the system to notify the user or designated contacts of specific information.

[1499] "Analysis results" refers to the information and evaluations obtained as a result of data analysis by a generative AI model.

[1500] A "certain threshold" refers to a reference value set based on the analysis results, and if this reference value is exceeded, a specific action will be taken.

[1501] "User" refers to the entity that uses the system, specifically, the individual or company that makes calls using the system.

[1502] "Trusted Contacts" refers to people or entities pre-defined by the user who are authorized to receive notifications regarding fraud and other emergencies.

[1503] "Anonymization" refers to techniques that remove or change personally identifiable information to protect the privacy of data.

[1504] "Storage" refers to techniques for storing data long-term and making it available for later reuse.

[1505] "Training data" refers to past data used as training material to improve the accuracy of generative AI models.

[1506] "Educational Materials" means written, audio, video, or other educational materials used to teach fraud prevention knowledge and techniques.

[1507] This invention is a system that records phone conversations, analyzes the recorded conversations using a generative AI model, determines the possibility of special fraud, and notifies the user. Specific embodiments for implementing this system are described in detail below.

[1508] System configuration

[1509] This system consists of a user's device, a server that stores and analyzes voice data, and a network for sending notifications.

[1510] Terminal

[1511] The device has the function of recording the contents of calls and sending the recorded data to a server in real time. When a call starts, the audio recording module automatically starts and buffers the audio data. This function is realized using a recording application on the smartphone or a dedicated call recording device.

[1512] server

[1513] The server receives the voice data sent from the device and analyzes it using a generative AI model. Specifically, a voice recognition module converts the voice data into text, and then analyzes the text data to determine whether it contains any signs of fraud. This analysis could use a machine learning algorithm (such as OpenAI's GPT-4 or Google BERT). Based on the analysis results, the server scores the likelihood of fraud, and if the score exceeds a certain threshold, it determines that there is a high possibility of fraud. In addition, the content of calls determined to be fraudulent is anonymized and stored as learning data or educational material.

[1514] Notification Network

[1515] The notification network notifies users and pre-defined trusted contacts when potential fraud is detected via text message, email, or dedicated in-app notifications, using the Twilio API or Firebase Cloud Messaging.

[1516] Specific examples

[1517] When user A receives a call from a self-proclaimed "bank representative" while on a call on his smartphone, the system operates as follows.

[1518] 1. When user A answers the phone, the device automatically records the call and sends the recording data to the server in real time.

[1519] 2. The server receives the voice data and the generative AI model analyzes it to confirm that the call contains keywords characteristic of fraud, such as "bank," "account number," and "transfer."

[1520] 3. The generative AI model generates a high fraud score (e.g., 93%). The server determines that there is a high probability of fraud and sends a notification.

[1521] 4. The server notifies User A and their trusted contacts via text message that a potentially fraudulent call was detected.

[1522] 5. The call transcripts are anonymized and stored in a fraud pattern database, which will be used to improve educational materials and AI models for future fraud prevention.

[1523] Example prompt sentence:

[1524] "Please convert the audio below to text and rate it as likely to be a scam. Please also be specific about which parts are characteristic of a scam.

[1525] Audio data: [Audio data URL]

[1526] This system enables early detection and rapid notification of toll fraud, as well as data accumulation and learning for fraud prevention.

[1527] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1528] Step 1:

[1529] A user starts a call on the device. The input is "start call." When the call starts, the device automatically starts the voice recording module and starts recording the contents of the call. The voice data is temporarily stored in a buffer. The output is the recorded "voice data of the call."

[1530] Step 2:

[1531] The device sends recorded audio data to the server in real time. The input is "call audio data." The device transfers the recorded file of the call content to the server via WebSocket or REST API. The output is "sent audio data" to the server.

[1532] Step 3:

[1533] The server receives the voice data and begins analysis using a generative AI model. The input is the "received voice data." The server converts the voice data into text using a speech recognition module. The text data is then input into a generative AI model (for example, OpenAI's GPT-4) for analysis. This detects whether the voice data contains features or patterns that indicate possible fraud. The output is the "analysis results."

[1534] Step 4:

[1535] The server scores the likelihood of fraud based on the analysis results. The input is the "analysis results." The generative AI model evaluates the likelihood of fraud based on prior learning and generates a score (e.g., a fraud score of 93%). The output is the "fraud score."

[1536] Step 5:

[1537] The server determines if the fraud score exceeds a certain threshold and prepares to send a notification if so. The inputs are the "fraud score" and the "threshold". In the case of a high fraud score above the threshold, the server generates a notification message and prepares to send it. The output is the "notification message".

[1538] Step 6:

[1539] The server sends notifications to the user and trusted contacts. The inputs are a "notification message" and "contact information." The server uses the Twilio API or Firebase Cloud Messaging to send notifications to inform the user and trusted contacts of possible fraud. The output is a "sent notification."

[1540] Step 7:

[1541] The server anonymizes the content of calls determined to be fraudulent and stores it in a pattern database. The input is "content of calls determined to be fraudulent." The server anonymizes the content of calls and stores it in a fraud pattern database for use as future learning data or educational material. The output is "stored anonymized data."

[1542] Step 8:

[1543] The server generates and distributes periodic reports to users and their families so that they can receive the latest information on fraud prevention. The inputs are "stored anonymized data" and "report distribution schedule." The server generates reports summarizing the latest fraud techniques and prevention strategies based on the stored data and distributes them periodically to users and trusted contacts. The output is "sent reports."

[1544] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1545] This invention is a system that records phone conversations, analyzes the recordings using a generative AI model and an emotion engine, determines the possibility of special fraud, and notifies the user if necessary. The following describes in detail an embodiment of this system.

[1546] System configuration

[1547] This system consists of the user's smartphone (device), a server that stores and analyzes voice data, a network for sending notifications, and an emotion engine that recognizes the user's emotions.

[1548] Terminal

[1549] The device has the function of recording the user's voice during a call and sending the data to the server in real time. When a call starts, the voice recording module automatically starts and buffers the voice data. The device is also equipped with a microphone and camera to capture the user's emotional data.

[1550] server

[1551] The server receives the voice and emotion data sent from the device and processes it in real time. The voice data is analyzed by a generative AI model to calculate a fraud probability score. The emotion engine also analyzes the user's emotional state.

[1552] Emotion Engine

[1553] The emotion engine is a module that recognizes the user's emotions from voice and image data. The emotion engine analyzes changes in voice tone and facial expressions to identify emotions such as nervousness, confusion, or surprise.

[1554] Notification Network

[1555] The notification network sends notifications to users and pre-defined trusted contacts (e.g., family members) when fraud is deemed likely, and can take emotional data into account to provide even more accurate warnings.

[1556] Program processing

[1557] Below we explain how each component of the system works.

[1558] Start recording a call (device)

[1559] When a user initiates a call on their smartphone, the device automatically records the call, storing the recording in a temporary buffer and sending it to a server in real time.

[1560] Receiving and analyzing voice data (server)

[1561] The server receives the voice data sent from the device and begins analysis using the generative AI model. The generative AI model converts the recorded data into text using a speech recognition module, and analyzes the text data to determine whether it contains any signs of fraud. As a result of the analysis, a score is assigned to the likelihood of fraud, and a final evaluation is made along with the user's emotional data using an emotion engine.

[1562] Acquisition and analysis of emotion data (device and server)

[1563] The device's microphone and camera capture the user's voice and facial expression data, which is then sent to a server that analyzes the data using an emotion engine to assess the user's emotional state. If the user expresses anxiety or confusion, this is also factored into the fraud likelihood score.

[1564] Possible fraud notification (server)

[1565] If a call is deemed likely to be fraudulent, the server sends a notification to pre-defined contacts (the user or one of their trusted contacts) informing them that a potentially fraudulent call has been detected and providing a warning based on the user's emotional state.

[1566] Storage and patterning of call content and emotion data (server)

[1567] If the call content and emotional data is determined to be potentially fraudulent, it is anonymized and stored in a database that records the characteristics of the fraud and the user's emotional state. This data will then be used to develop fraud prevention education materials and improve future AI models. Users and their families will receive regular updates on the latest fraud scams and prevention strategies.

[1568] Specific examples

[1569] System Operation Scenarios

[1570] For example, if user B receives a call from a self-proclaimed "police officer" while on a call on his smartphone, the system operates as follows.

[1571] 1. When User B answers the phone, the device automatically records the call and simultaneously captures emotion data.

[1572] 2. The recorded data and emotional data are sent to the server in real time.

[1573] 3. The generative AI model analyzes the voice data and detects fraud keywords such as "police" and "ID." In addition, the emotion engine detects User B's anxious voice and confused facial expression.

[1574] 4. Based on a high fraud score (e.g., 90%) and User B's emotional state, fraud is deemed more likely.

[1575] 5. The server sends a text message to User B and his family saying, "A likely fraudulent call has been detected. User B is in a state of anxiety."

[1576] 6. The call content and emotional data is anonymized and stored in a fraud pattern database.

[1577] 7. User B and his family will receive a report at a later date containing updated information on fraud prevention.

[1578] This system will enable more accurate fraud detection based on the user's emotional state, providing quick and accurate protection from fraud. As a result, it will provide an environment where all users, including the elderly, can make calls with peace of mind.

[1579] The processing flow will be explained below.

[1580] Step 1: Start recording the call (on your device)

[1581] When a user starts a call on the smartphone, the terminal detects a call start event.

[1582] The device will automatically start the audio recording module.

[1583] The device stores the recorded data in a temporary buffer and prepares it for transmission to the server in real time.

[1584] Step 2: Receiving audio data (server)

[1585] The server receives the voice data packets sent from the terminal.

[1586] The received audio data is stored in a temporary storage device.

[1587] The server queues the audio data for analysis by the generative AI model.

[1588] Step 3: Analyzing the voice data (server)

[1589] The server retrieves the audio data from the analysis queue.

[1590] The server uses voice recognition technology to convert the voice data into text data.

[1591] The server uses a generative AI model to analyze the text data and detect fraudulent features.

[1592] Based on the analysis results, a fraud likelihood score is calculated.

[1593] Step 4: Acquiring emotion data (device)

[1594] The device's microphone and camera capture the user's voice and facial expression data.

[1595] The terminal transmits audio and image data to the server in real time.

[1596] Step 5: Emotion data analysis (server)

[1597] The server uses an emotion engine to analyze the received audio and image data.

[1598] An emotion engine identifies the user's emotional state (e.g., anxiety, confusion, surprise, etc.).

[1599] The analysis results of the emotion data and voice data are combined to update the final fraud likelihood score.

[1600] Step 6: Determine the likelihood of fraud (server)

[1601] The server evaluates the updated fraud likelihood score.

[1602] Determine whether the score exceeds a set threshold.

[1603] If fraud is deemed likely, prepare to notify.

[1604] Step 7: Sending Notifications (Server)

[1605] The server generates a notification message if fraud is deemed likely.

[1606] The notification includes information about the likelihood of fraud and the user's emotional state.

[1607] Notifications will be sent to the user and pre-registered contacts (such as family members) via text message or in-app notification.

[1608] Step 8: Storing and patterning call content and emotion data (server)

[1609] The server anonymizes the content of calls that are deemed likely to be fraudulent.

[1610] The anonymized data, including emotional data, is stored in a fraud pattern database.

[1611] The latest fraud methods are analyzed based on the collected data.

[1612] Regularly provide fraud prevention educational information to you and your family.

[1613] Example 2

[1614] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1615] Conventional call recording systems have low detection accuracy for special frauds, making it difficult to prevent fraud victims. Furthermore, because they do not take into account the user's emotional state, they are unable to properly assess the anxiety and confusion felt by the user. Furthermore, when a potential fraud is detected, notification methods are limited, making it impossible to quickly communicate warnings to a wide range of contacts. These issues need to be resolved.

[1616] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1617] In this invention, the server includes a means for recording the content of the call, a means for analyzing the recorded content of the call using a generative AI model, a means for determining the possibility of special fraud based on the analysis results, a means for acquiring and analyzing emotional data, and a means for sending a notification if there is a possibility of special fraud. This allows for simultaneous analysis of the content of the call and the user's emotional state, enabling more accurate fraud detection and faster notification.

[1618] A "call recording device" is a device or software that allows a user to capture and save audio data in real time while making a call.

[1619] "Means of analysis using a generative AI model" refers to artificial intelligence technology that converts recorded audio data into text data and analyzes the characteristics of fraud.

[1620] "Means for determining the possibility of special fraud" refers to algorithms or programs for evaluating and scoring the possibility of fraud based on the analysis results of the generative AI model.

[1621] "Means for acquiring and analyzing emotional data" refers to emotion recognition technology for capturing a user's vocal tone and facial expressions and analyzing the data to assess the user's emotional state.

[1622] "Means for sending notifications" refers to communication means for sending information to the user and pre-defined trusted contacts if fraud is determined to be likely.

[1623] This invention is a system that records phone conversations, analyzes the recordings using a generative AI model and an emotion engine, determines the possibility of special fraud, and notifies the user if necessary. The following describes in detail an embodiment of this system.

[1624] System configuration

[1625] This system consists of the user's smartphone (device), a server that stores and analyzes voice data, a network for sending notifications, and an emotion engine that recognizes the user's emotions.

[1626] Terminal

[1627] The device has the function of recording the user's voice during a call and sending the data to the server in real time. When a call starts, the voice recording module automatically starts and buffers the voice data. The device is also equipped with a microphone and camera to capture the user's emotional data.

[1628] server

[1629] The server receives the voice and emotion data sent from the device and processes it in real time. The voice data is analyzed by a generative AI model to calculate a fraud probability score. The emotion engine also analyzes the user's emotional state.

[1630] Emotion Engine

[1631] The emotion engine is a module that recognizes the user's emotions from voice and image data. The emotion engine analyzes changes in voice tone and facial expressions to identify emotions such as nervousness, confusion, or surprise.

[1632] Notification Network

[1633] The notification network sends notifications to users and pre-defined trusted contacts (e.g., family members) when fraud is deemed likely, and can take emotional data into account to provide even more accurate warnings.

[1634] Program processing

[1635] Below we explain how each component of the system works.

[1636] Start recording a call (device)

[1637] When a user starts a call on their smartphone, the device automatically records the conversation. This recording is temporarily stored in a buffer and then sent to a server in real time. The device is also equipped with a microphone and camera to capture the user's emotional data.

[1638] Receiving and analyzing voice data (server)

[1639] The server receives the voice data sent from the device and begins analysis using the generative AI model. Specifically, it uses a voice recognition module to convert the recorded data into text data, and then analyzes the text data to determine whether it contains any fraudulent characteristics. This analysis calculates a fraud likelihood score. The emotion engine simultaneously analyzes the user's emotional data, and a final evaluation is made.

[1640] Acquisition and analysis of emotion data (device and server)

[1641] The device's microphone and camera capture the user's voice and facial expression data, which is then sent to a server that analyzes it using an emotion engine to assess the user's emotional state. If the user expresses anxiety or confusion, this is also factored into the fraud likelihood score.

[1642] Possible fraud notification (server)

[1643] If a fraudulent call is determined to be likely, the server sends a notification to a pre-defined contact with the message, "A fraudulent call has been detected. User B is in a state of anxiety."

[1644] Storage and patterning of call content and emotion data (server)

[1645] If a call is deemed potentially fraudulent, the call's content and emotional data will be stored in an anonymized database. This data will then be used to improve fraud prevention education materials and AI models. Users and their families will also receive regular updates on the latest fraud scams and prevention strategies.

[1646] Example: System Operation Scenarios

[1647] For example, if a user receives a call from a self-proclaimed "police officer" while on a call on their smartphone, the system operates as follows.

[1648] When the user answers the call, the device automatically records the call and simultaneously captures emotional data.

[1649] Recorded data and emotional data are sent to the server in real time.

[1650] A generative AI model analyzes the audio data to detect fraud keywords such as "police" and "ID," while an emotion engine detects anxious voices and confused facial expressions.

[1651] A high fraud score (e.g., 90%) and the user's emotional state are evaluated to determine the likelihood of fraud.

[1652] The server sends a text message to the user and their family saying, "A likely fraudulent call has been detected. The user is in a state of anxiety."

[1653] The call content and emotional data is anonymized and stored in a fraud pattern database.

[1654] Users and their families will receive a report at a later date containing updated information on fraud prevention.

[1655] Prompt Sentence Examples

[1656] "Specify how a system works to detect potential fraud while a user is on a call."

[1657] This system will enable more accurate fraud detection based on the user's emotional state, providing quick and accurate protection from fraud. As a result, it will provide an environment where all users, including the elderly, can make calls with peace of mind.

[1658] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1659] Step 1:

[1660] When a user initiates a call on their smartphone, the device automatically records the call, temporarily stores the recording in a buffer, and then transmits it to a server in real time.

[1661] Specific behavior:

[1662] 1. When a user makes or receives a call, the smartphone's voice recording module is automatically activated.

[1663] 2. Audio data is captured through the built-in microphone and temporarily stored in the device's memory.

[1664] 3. The buffered data is encrypted and sent to the server in real time.

[1665] Input: Call audio data

[1666] Output: Encrypted recording data

[1667] Step 2:

[1668] The server receives the voice data sent from the device and temporarily stores the recorded data in a database, while simultaneously starting analysis using the generative AI model.

[1669] Specific behavior:

[1670] 1. The server receives the encrypted voice data, decrypts it, and temporarily stores it in a database.

[1671] 2. The decoded voice data is input into the generative AI model and converted into text data using a voice recognition module.

[1672] 3. A generative AI model analyzes the text data and calculates a fraud likelihood score.

[1673] Input: Encrypted audio recording

[1674] Output: Analysis results including fraud likelihood score

[1675] Step 3:

[1676] The device's microphone and camera capture the user's voice and facial expression data, which is then sent to a server.

[1677] Specific behavior:

[1678] 1. The device captures the user's voice tone with the microphone.

[1679] 2. The camera captures the user's facial expressions in real time.

[1680] 3. The acquired audio and image data are sent to the server.

[1681] Input: Voice tone and facial expression data

[1682] Output: Voice tone and facial expression data sent to the server

[1683] Step 4:

[1684] The server receives the emotion data sent from the terminal and analyzes it using an emotion engine.

[1685] Specific behavior:

[1686] 1. The server inputs the received voice tone and facial expression data into the emotion engine.

[1687] 2. The emotion engine analyzes voice tone and facial expressions to assess the user's emotional state.

[1688] 3. The emotional state assessment results are combined with the deception likelihood score to form a final assessment.

[1689] Input: Voice tone and facial expression data

[1690] Output: Evaluation results including the user's emotional state

[1691] Step 5:

[1692] If fraud is deemed likely, the server will send a notification to pre-defined contacts.

[1693] Specific behavior:

[1694] 1. The server makes a comprehensive assessment based on the fraud probability score and the user's emotional state.

[1695] 2. Generate a notification message if fraud is determined to be likely.

[1696] 3. Send a notification message to you and your pre-defined trusted contacts.

[1697] Input: Evaluation results including the user's emotional state

[1698] Output: Notification message

[1699] Step 6:

[1700] The server anonymizes the content of calls that are deemed likely to be fraudulent and the user's emotional data and stores them in a database.

[1701] Specific behavior:

[1702] 1. Anonymize data that the analysis assesses as likely to be fraudulent.

[1703] 2. Store the anonymized data in a fraud pattern database.

[1704] 3. The data will later be used to improve fraud prevention educational materials and AI models.

[1705] Input: Call content and emotion data including evaluation results

[1706] Output: Anonymized database records

[1707] (Application example 2)

[1708] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1709] In today's society, the number of victims of telephone fraud is increasing, and elderly people in particular are prone to falling prey to the sophisticated tactics of fraudsters. Furthermore, conventional fraud prevention systems have issues with their difficulty in accurately identifying fraud and their inability to issue warnings at the appropriate time because they do not take the user's emotional state into account. This invention aims to provide a system that can accurately detect the possibility of fraud and issue warnings at the appropriate time by analyzing the content of phone calls and comprehensively assessing the user's emotional state.

[1710] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1711] In this invention, the server includes a means for recording call content, a means for analyzing the recorded call content with a generative AI model, a means for detecting fraud keywords, a means for acquiring and analyzing user emotion data, a means for scoring the likelihood of fraud based on the analysis results and the emotion data, and a means for sending a notification to the user and trusted contacts if the likelihood of fraud is high, thereby enabling advanced fraud detection and warning based not only on call content but also on the user's emotional state.

[1712] "Call content" refers to the audio and text data exchanged during a phone or video call.

[1713] "Recording" is the process of storing telephone conversations in digital or analog form.

[1714] A "generative AI model" is an artificial intelligence model that uses machine learning algorithms to analyze voice data and identify specific patterns and keywords.

[1715] "Analysis" is the process of evaluating recorded data to detect specific patterns or characteristics.

[1716] "Fraud keywords" are specific words or phrases used to characterize fraudulent activity.

[1717] "Emotion data" refers to data that includes changes in voice tone and facial expressions that indicate the emotional state of the user during a call.

[1718] "Scoring" is the process of evaluating data based on specific criteria and assigning them a score accordingly.

[1719] "Notification" is a communication method used to convey alerts or information to users or trusted contacts.

[1720] "Trusted Contacts" refers to people or organizations that a user pre-selects with whom they can share emergency or important information.

[1721] "Anonymization" is the process of removing personally identifiable information from data so that it cannot be identified by third parties.

[1722] A "pattern" refers to an arrangement or tendency of a set of data based on a particular characteristic or feature.

[1723] The system that realizes this invention is made up of a complex configuration including a user terminal, a server, a notification network, and an emotion engine.

[1724] Terminal

[1725] The device is assumed to be a smartphone used by the user. This smartphone is equipped with a voice recording module for recording the contents of calls. When a call is initiated, recording starts automatically and the voice data is sent to a server in real time. The device is also equipped with a microphone and a camera, which can capture the user's emotional data.

[1726] server

[1727] The server plays a central role in processing the voice and emotion data sent from the devices. The following software is installed on the server:

[1728] Generative AI model: The generative AI model uses a speech recognition module to convert the recording into text, then analyzes it for potential fraud. If fraud keywords are detected, the analysis results are scored.

[1729] Emotion Engine: The emotion engine analyzes voice tone and facial expression data to assess the user's emotional state. If the user is anxious or confused, this will also be reflected in the fraud likelihood score.

[1730] Notification Network

[1731] The server assesses the likelihood of fraud based on the results of the generative AI model and emotion engine analysis, and sends notifications to users and pre-defined trusted contacts if there is a high likelihood of fraud. The network can issue alerts via text message or in-app notification.

[1732] Specific examples

[1733] For example, if a user receives a phone call from someone who identifies themselves as a "police officer," the modules in the system will act as follows:

[1734] 1. When a user receives a call, the device's voice recording module automatically starts recording and transmits the voice data to the server in real time.

[1735] 2. A generative AI model is installed on the server, and fraud keywords such as "police" and "ID" are detected from the voice data.

[1736] 3. At the same time, the device's microphone and camera capture the user's voice tone and facial expression data, which are then analyzed by the emotion engine. It can be seen that the user's voice sounds anxious or their facial expression looks confused.

[1737] 4. Based on the analysis results of the generative AI model and the emotion engine, a high fraud score is calculated. Specifically, the generative AI model's analysis result indicates a fraud probability score of 90%, and the emotion engine's analysis result also determines that the user is in an anxious state.

[1738] 5. If the final evaluation results in a high fraud score (e.g., 90%), the server sends a notification to the user and trusted contacts stating, "A potentially fraudulent call has been detected."

[1739] An example prompt is:

[1740] "The user answers the phone and starts talking. At the same time as the voice data, facial expression data is also captured and sent to the server. Keywords such as 'police,' 'bank transfer,' and 'ID card' are detected on the server side, and at the same time the user's face shows signs of confusion or anxiety. The system rates these with a high score and notifies the user and their trusted contacts that 'A call with a high probability of fraud has been detected.'"

[1741] This allows users to be quickly warned of potential fraud and prevent it from happening. The system utilizes cutting-edge artificial intelligence and emotion recognition technology to provide an effective means of protecting users' safety in real time.

[1742] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1743] Step 1:

[1744] Start recording a call

[1745] When a user starts a call on their smartphone, the voice recording module is automatically started. The recorded voice data is temporarily stored in a buffer. In this step, the input is the call start trigger and the output is the recorded voice data.

[1746] Step 2:

[1747] Acquiring voice and emotion data

[1748] The microphone and camera on the device capture the user's voice tone and facial expression data. These data are sent to the server in real time in parallel. The input is voice and facial expression capture, and the output is voice data and emotional data.

[1749] Step 3:

[1750] Analysis of audio data

[1751] The server analyzes the received voice data using a generative AI model. The voice data is first converted into text, and fraud keywords are detected from the text data. The input is the voice data, and the output is the analyzed text data and fraud keywords.

[1752] Step 4:

[1753] Emotional Data Analysis

[1754] The server analyzes the received emotional data using an emotion engine. It evaluates the user's emotional state based on the voice tone and facial expression data. In this step, the input is the emotion data, and the output is the analyzed emotional state.

[1755] Step 5:

[1756] Fraud likelihood scoring

[1757] The server scores the likelihood of fraud based on the analysis results of the generative AI model and the analysis results of the emotional data by the emotion engine. The fraud keyword detection results are combined with the user's emotional state to calculate an overall score. The inputs are text data and emotional state, and the output is a fraud likelihood score.

[1758] Step 6:

[1759] Sending notifications

[1760] If the fraud likelihood score is high, the server will send a notification to the user and pre-defined trusted contacts. The notification will include a notification that a fraud likelihood call was detected and a warning based on the analysis results. The input is the fraud likelihood score and the output is the warning notification.

[1761] Step 7:

[1762] Call content and emotion data storage and patterning

[1763] The content and emotion data of calls that are judged to be highly likely to be fraudulent are anonymized and stored. This creates a database for future fraud prevention. The inputs are the call data and emotion data that are judged to be fraudulent, and the output is an updated fraud prevention pattern database.

[1764] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1765] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1766] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1767] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1768] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1769] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1770] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1771] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1772] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1773] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1774] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1775] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1776] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1777] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1778] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1779] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1780] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1781] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1782] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1783] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1784] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1785] The following is further disclosed regarding the above embodiment.

[1786] (Claim 1)

[1787] A means for recording the contents of a call;

[1788] A means to analyze the recorded call content using a generative AI model,

[1789] A means of determining the possibility of special fraud based on the analysis results, and

[1790] A means of sending notifications in the event of a possible special fraud;

[1791] A system including:

[1792] (Claim 2)

[1793] 10. The system of claim 1, further comprising means for scoring the likelihood of fraud based on the analysis results.

[1794] (Claim 3)

[1795] 10. The system of claim 1, further comprising means for anonymizing and storing the call content and recording it as a pattern when possible fraud is detected.

[1796] "Example 1"

[1797] (Claim 1)

[1798] means for automatically recording a call when the user initiates the call;

[1799] means for transmitting the recorded audio data to a server in real time;

[1800] means for converting received voice data into text data using voice recognition technology;

[1801] a means for analyzing the converted text data using a generative AI model to determine likelihood of fraud;

[1802] a means of sending notifications to users and designated third parties in the event of possible fraud;

[1803] A means of anonymizing and storing potentially fraudulent calls and recording them in a fraud pattern database;

[1804] A system including:

[1805] (Claim 2)

[1806] 10. The system of claim 1, further comprising means for scoring the likelihood of fraud based on the analysis results using a generative AI model.

[1807] (Claim 3)

[1808] The system of claim 1 further comprising a means for anonymizing and storing call content determined to be highly likely to be fraudulent in a pattern database for use in future AI model improvements and fraud prevention education.

[1809] "Application Example 1"

[1810] (Claim 1)

[1811] A means for recording the contents of a call;

[1812] A means to analyze the recorded call content using a generative AI model,

[1813] A means of determining the possibility of special fraud based on the analysis results, and

[1814] A means of sending notifications in the event of a possible special fraud;

[1815] means for sending notifications to the user and trusted contacts when the analysis results exceed a certain threshold;

[1816] The call content will be anonymized and saved for later use as training data and educational materials for fraud prevention.

[1817] A system including:

[1818] (Claim 2)

[1819] 10. The system of claim 1, further comprising means for scoring the likelihood of fraud based on the analysis results.

[1820] (Claim 3)

[1821] 10. The system of claim 1, further comprising means for anonymizing and storing the call content and recording it as a pattern when possible fraud is detected.

[1822] "Example 2: Combining Emotion Engines"

[1823] (Claim 1)

[1824] A means for recording the contents of a call;

[1825] A means to analyze the recorded call content using a generative AI model,

[1826] A means of determining the possibility of special fraud based on the analysis results, and

[1827] a means for acquiring and analyzing emotion data;

[1828] A means of sending notifications in the event of a possible special fraud;

[1829] A system including:

[1830] (Claim 2)

[1831] 10. The system of claim 1, further comprising means for scoring the likelihood of fraud based on the analysis results.

[1832] (Claim 3)

[1833] 10. The system of claim 1, further comprising means for anonymizing and storing the call content and recording it as a pattern when possible fraud is detected.

[1834] "Application example 2 when combining emotion engines"

[1835] (Claim 1)

[1836] A means for recording the contents of a call;

[1837] A means to analyze the recorded call content using a generative AI model,

[1838] A means of determining the possibility of special fraud based on the analysis results, and

[1839] a means for detecting fraudulent keywords;

[1840] means for acquiring and analyzing user emotion data;

[1841] means for scoring the likelihood of fraud based on the analysis results and sentiment data;

[1842] means for sending notifications to the user and trusted contacts in the event of a possible special fraud;

[1843] A system including:

[1844] (Claim 2)

[1845] 10. The system of claim 1, further comprising means for scoring the likelihood of fraud based on the analysis results.

[1846] (Claim 3)

[1847] 10. The system of claim 1, further comprising means for anonymizing and storing the call content and recording it as a pattern when possible fraud is detected. [Explanation of symbols]

[1848] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for recording the contents of a call; A means to analyze the recorded call content using a generative AI model, A means of determining the possibility of special fraud based on the analysis results, and A means of sending notifications in the event of a possible special fraud; A system including:

2. The system of claim 1 further comprising means for scoring the likelihood of fraud based on the analysis results.

3. 10. The system of claim 1, further comprising means for anonymizing and storing the content of the call and recording it as a pattern when possible fraud is detected.

Images