System

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting audio data; means for converting the collected audio data into textual information; generative artificial intelligence means for detecting fraud-related patterns using the converted textual information; and means for sending an alert when a fraud-related pattern is detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In modern society, the number of victims of deposit fraud, particularly targeting the elderly, is rapidly increasing. Such fraud not only causes financial loss to victims, but also psychological distress. Existing fraud prevention measures mainly rely on advance warnings, and lack the means to detect the moment a fraud actually occurs in real time and prevent it before it occurs. Therefore, there is a need for a system that can monitor and detect potentially fraudulent conversations in real time and immediately issue an alert. [Means for solving the problem]

[0005] We provide a system that includes a means for collecting voice data, a means for converting the collected voice data into text information, a generative artificial intelligence means for detecting fraud-related patterns using the converted text information, and a means for sending an alert when a fraud-related pattern is detected. This system uses generative artificial intelligence to detect possible fraud with high accuracy by comparing the data with a huge data set of past frauds in real time. It also incorporates a function that immediately sends an alert to family members and the police when a fraud-related pattern is detected, preventing fraud victims from becoming victims.

[0006] "Audio data" refers to data that has been recorded and saved in digital format, including audio signals such as the user's voice and environmental sounds.

[0007] "Collection means" refers to hardware and software that acquires audio data from the environment using microphones, voice recognition sensors, etc.

[0008] "Means for converting into textual information" refers to a process or system that analyzes audio data and converts it into corresponding textual data.

[0009] "Generative AI tools" refer to algorithms and systems that use artificial intelligence techniques to generate data and detect specific patterns or features.

[0010] "Fraud-related patterns" are specific words, phrases, conversational flow, and other characteristics that are frequently used when fraudulent activity occurs.

[0011] "Means for sending an alert" means hardware and software for sending a notification to pre-designated contacts when a fraud-related pattern is detected.

[0012] A "past fraud dataset" is a database that collects and organizes data on various fraud cases that have occurred in the past.

[0013] "Family and Law Enforcement" refers to the user's relatives and legal organizations that should be registered in advance to prevent and respond to fraud. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0015] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0018] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0019] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0020] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0022] [First embodiment]

[0023] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0024] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0027] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0030] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0034] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0035] 1. Collection of audio data

[0036] The device constantly monitors surrounding voice data using a microphone. The device also has a voice recognition sensor that collects user conversations in real time. For example, voice data is acquired when the user is talking on the phone or talking face-to-face with family members.

[0037] 2. Transcription of audio data

[0038] The device uses software to convert the collected voice data into text. This software uses speech recognition technology to analyze the voice data and generate corresponding text. For example, if a user says, "Is this from my bank?", this phrase will be recorded as text.

[0039] 3. Fraud Pattern Detection

[0040] The server receives text sent from the device and uses generative artificial intelligence to detect fraud-related patterns. This AI model is trained on historical fraud datasets and determines in real time whether the user's conversation matches the characteristics of fraud. For example, if a phrase like "Can you tell me my account information?" is detected in the conversation, it will mark it as a possible fraud.

[0041] 4. Sending alerts

[0042] If a fraud-related pattern is detected, the server immediately sends an alert. Recipients of the alert include pre-registered family members and police agencies. For example, if a user utters a typical fraudulent phrase like "Do I need to provide my account number?", the server will send an alert message to family members and the police saying, "A possible fraudulent conversation has been detected."

[0043] Specific examples

[0044] User Status

[0045] The user is an elderly person who regularly communicates with his family and bank over the phone. One day, he receives a call saying, "Hello, is this the bank? I'd like to check my account information."

[0046] Device behavior

[0047] The terminal that receives this conversation immediately collects the voice data and converts it into text information.

[0048] Server Operation

[0049] The user's conversation is sent to a server, which uses generative artificial intelligence to detect fraud patterns. In this case, the phrase "I would like to check my account information" matches the fraud pattern, so it is immediately determined to be a potential fraud.

[0050] Sending alerts

[0051] If a fraudulent conversation is detected, the server sends a warning message to the user's family and the police, saying, "A potential fraudulent conversation has been detected." This allows the family and the police to respond quickly and prevent the user from becoming a victim of fraud.

[0052] The above is a specific embodiment for carrying out the present invention. The present invention protects users from the risk of fraud and allows them to quickly receive support from their families or the police.

[0053] The processing flow will be explained below.

[0054] Specific steps for collecting and processing audio data

[0055] Step 1: Collecting audio data

[0056] The device activates voice recognition software and captures surrounding audio through a microphone, monitoring all user conversations in real time while filtering out background noise and other distracting sounds.

[0057] Step 2: Temporarily save the audio data

[0058] The device temporarily stores the collected voice data in its internal memory. This data is temporarily held for subsequent processing. As voice data is collected, it is accumulated in a buffer.

[0059] Step 3: Transcribe the audio data

[0060] The device uses voice recognition software to convert the temporarily stored voice data into text. The text generated by this process is a textual representation of what the user said. For example, "Hello, is this from my bank?"

[0061] Step 4: Send text information

[0062] The device then sends the converted text data, including necessary metadata (such as a timestamp and user ID), to a server via the Internet.

[0063] Step 5: Detect fraud patterns

[0064] The server analyzes the received text and uses generative artificial intelligence to detect fraud patterns. Specifically, it compares it with historical data sets and identifies phrases and wording characteristic of fraud. This analysis is done in real time.

[0065] Step 6: Determine the likelihood of fraud

[0066] The server evaluates the likelihood of fraud based on the results of the generative AI. If the algorithm determines that there is a high risk of fraud, the result is recorded.

[0067] Step 7: Building the Alert

[0068] If the server detects possible fraud, it constructs an alert message that includes the nature of the detected fraud, the date and time, and the affected user.

[0069] Step 8: Sending an alert

[0070] The server sends alert messages to pre-registered family members and police agencies via email and SMS, allowing recipients to receive the alerts immediately and respond quickly.

[0071] Step 9: Logging

[0072] The server records detailed logs of all operations it performs, including timestamps of operations, fraud patterns detected, and details of warning messages sent, for later analysis and troubleshooting.

[0073] Example 1

[0074] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0075] Today, the risk of becoming a victim of fraud is increasing, especially among the elderly, and there is a need to detect signs of fraud in real time and respond quickly. Furthermore, as fraud methods become more sophisticated, it is difficult to respond with traditional manual monitoring and vigilance, and cooperation from family members and police agencies is essential. Therefore, it is necessary to realize a fraud detection system that uses voice data.

[0076] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0077] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text information, means for transmitting the converted text information to the server, generative artificial intelligence means for detecting fraud-related patterns using the text information received by the server, and means for sending an alert when a fraud-related pattern is detected. This makes it possible to detect signs of fraud in real time, send an alert quickly, and protect users from fraud.

[0078] "Voice data" refers to information that is a digital recording of the user's speech and surrounding sounds.

[0079] "Collection means" refers to a device or method for acquiring audio data using a microphone, sensor, etc.

[0080] "Text information" refers to text data converted from collected voice data using voice recognition technology.

[0081] "Means of converting into textual information" refers to the process of converting audio data into textual data using speech recognition software or algorithms.

[0082] The "means for transmitting the converted character information" is a mechanism for transferring the generated text data to a server via a secure protocol.

[0083] "Generative artificial intelligence" is an AI technology that learns from past data sets and makes predictions and judgments in real time about new data.

[0084] "Fraud-related patterns" refer to specific phrases or patterns of behavior that indicate fraudulent activity.

[0085] "Means for sending an alert" refers to a method for sending a warning message to pre-registered recipients when a fraud-related pattern is detected.

[0086] A "historical fraud dataset" is a collection of previously recorded fraud cases and patterns.

[0087] "Family and Law Enforcement" refers to contacts that include the user's pre-registered relatives and law enforcement agencies.

[0088] MODE FOR CARRYING OUT THE INVENTION

[0089] The present invention relates to a system that uses specific hardware and software to provide real-time fraud protection to users, and a specific embodiment of the system is described below.

[0090] Audio data collection

[0091] The device constantly collects surrounding audio data using a microphone. The device is equipped with a high-performance voice recognition sensor that captures the user's conversation in real time. The device can be a smartphone, tablet, smart speaker, or other device. For example, when a user is talking on the phone, the device can be equipped with a function to record the conversation.

[0092] Transcription of audio data

[0093] The device uses speech recognition software to convert collected voice data into text. Specifically, it uses speech recognition technologies such as the Google Speech-to-Text API and Microsoft Azure Cognitive Services Speech API. This software analyzes the voice data and generates corresponding text. For example, if a user says, "Is this from my bank?", this phrase will be recorded as text on the device.

[0094] Sending audio data

[0095] The device sends the generated text data to the server in an encrypted format using a secure protocol (e.g., HTTPS), which ensures data security and privacy.

[0096] Fraud pattern detection

[0097] The server receives the text sent from the device and begins analyzing it in real time using generative artificial intelligence (e.g., OpenAI GPT). The software on the server is trained on historical fraud datasets and determines whether the text matches the characteristics of fraud. For example, if the phrase "Please tell me your account information" is detected, it will be marked as a possible fraud.

[0098] Sending alerts

[0099] If a fraud-related pattern is detected, the server immediately sends an alert. Recipients of the alert include pre-registered family members and police agencies. For example, if a user says, "Do I need to provide my account number?", the server will send an alert message to family members and the police saying, "A potentially fraudulent conversation has been detected."

[0100] Specific examples

[0101] User Status

[0102] The user is an elderly person who regularly communicates with family and financial institutions over the phone. One day, he receives a call saying, "Hello, is this the bank? I'd like to check my account information."

[0103] Device behavior

[0104] The device that receives this conversation immediately collects the voice data and converts it into text information. The device is a smartphone, and the voice data is converted into text using the Google Speech-to-Text API.

[0105] Server Operation

[0106] The user's conversation is sent to a server, which uses generative artificial intelligence to detect fraud patterns. In this case, the phrase "I would like to check my account information" matches the fraud pattern, so it is immediately determined to be a potential fraud.

[0107] Sending alerts

[0108] If a fraudulent conversation is detected, the server sends a warning message to the user's family and the police, saying, "A potential fraudulent conversation has been detected." This allows the family and the police to respond quickly and prevent the user from becoming a victim of fraud.

[0109] Prompt Sentence Examples

[0110] "If a user calls and says, 'Is this your bank? I'd like to confirm my account information,' we'll mark it as a fraud pattern and send an alert message to their family and the police."

[0111] This explains the configuration of a system for specifically implementing the present invention, which protects users from the risk of fraud and sends prompt alerts to obtain cooperation from family members and the police.

[0112] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0113] Step 1:

[0114] The device constantly collects surrounding audio data using a microphone. When the user speaks, the device captures the audio and records it in digital form. The input is the user's speech, and the output is digital audio data. For example, a user might record themselves saying, "Hello, is this the bank?"

[0115] Step 2:

[0116] The device converts the collected voice data into text using the Google Speech-to-Text API. During this conversion process, voice recognition technology is used to analyze the voice data and generate corresponding text. The input is digital voice data, and the output is text data. Specifically, the voice saying "Hello, is this the bank?" is converted into text "Hello, is this the bank?"

[0117] Step 3:

[0118] The terminal securely transmits the generated text data to the server using the HTTPS protocol. During the data transmission process, data security and privacy are guaranteed by encryption. The input is text data, and the output is secure data transmission to the server. Specifically, the text data "Hello, is this the bank?" is transmitted to the server.

[0119] Step 4:

[0120] The server analyzes the received text data in real time. It runs a generative artificial intelligence (e.g., OpenAI GPT) and compares it with past fraud datasets to detect fraud-related patterns. The input is text data, and the output is the detection result of fraud-related patterns. For example, if the phrase "Please tell me your account information" is detected, it is marked as a possible fraud.

[0121] Step 5:

[0122] If a fraud-related pattern is detected, the server immediately generates an alert and sends it to pre-registered family members and police agencies. During this process, the generated alert message is forwarded to each contact in the distribution list. The input is a fraud-related text phrase, and the output is the alert message sent to family members and police agencies. Specifically, the alert message sent is "A potential fraudulent conversation has been detected."

[0123] Through these steps, the system protects users from the risk of fraud and can quickly issue alerts to prompt action from relevant authorities.

[0124] (Application example 1)

[0125] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0126] In modern society, cases of elderly people falling victim to telephone fraud are increasing. To prevent such fraud, real-time fraud detection and rapid response methods are required. However, conventional methods are insufficient to efficiently monitor telephone conversations and detect possible fraud, making it difficult to completely prevent fraud. Technology to address this issue is needed.

[0127] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0128] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text information, generative artificial intelligence means for detecting fraud-related patterns using the converted text information, means for sending an alert when a fraud-related pattern is detected, means for converting voice data into text using voice recognition software, means for analyzing fraud patterns using a generative artificial intelligence model using a cloud service, and notification means for sending an emergency notification when fraud is detected. This makes it possible to detect potentially fraudulent telephone conversations in real time and quickly notify family members and the police.

[0129] "Audio data" refers to information that records audio in digital format.

[0130] "Means of collection" refers to devices and software for acquiring and storing voice data.

[0131] "Means for converting into textual information" refers to technology or software that converts audio data into text format.

[0132] "Generative artificial intelligence" refers to an AI model that learns from past data and analyzes new data.

[0133] "Detecting fraud-related patterns" is the process of determining whether certain phrases or behaviors are associated with fraud.

[0134] An "alert means" is a system or technology for providing notification in the event of suspected fraud.

[0135] "Speech recognition software" is a program that converts speech into text in real time.

[0136] "Cloud services" are computer resources and data analysis services provided via the Internet.

[0137] A "generative artificial intelligence model" is an AI model used to detect fraud patterns from text data.

[0138] A "notification mechanism" is a system capable of sending emergency notifications based on detected fraud patterns.

[0139] This invention is a real-time fraud detection and emergency notification system that monitors users' phone conversations using their smartphones and can take immediate action if fraudulent activity is suspected.

[0140] System configuration

[0141] The system consists of the following main components:

[0142] 1. User device (smartphone)

[0143] 2. Speech recognition software (Google Speech-to-Text API)

[0144] 3. Cloud Services (Amazon Web Services: AWS Lambda and AWS SNS)

[0145] 4. Generative AI models (e.g., GPT-4)

[0146] Hardware and Software Use

[0147] The user device is equipped with a built-in microphone that is used to collect real-time phone conversations, and this voice data is converted into text using the Google Speech-to-Text API.

[0148] The converted text is sent to AWS Lambda, a cloud service, and analyzed by a generative AI model (e.g., GPT-4), which is trained on historical fraud datasets to detect phrases and patterns characteristic of fraud.

[0149] If a suspicious pattern is detected, AWS SNS will send an emergency notification to pre-registered family members and law enforcement agencies, including a message such as "A potential fraudulent conversation has been detected."

[0150] Specific examples of processing

[0151] User Status

[0152] The user is an elderly person who regularly communicates with family, banks, etc. by telephone. One day, he receives a call saying, "Hello, this is the bank. I would like to confirm my account information."

[0153] Device behavior

[0154] The user's device collects this conversation in real time and converts the audio data into text using the Google Speech-to-Text API.

[0155] Server Operation

[0156] The converted text is sent to AWS Lambda in the cloud and analyzed using a generative artificial intelligence model (e.g., GPT-4). If the detected patterns match the characteristics of a fraud, an emergency notification is sent to family members and police via AWS SNS.

[0157] Prompt Sentence Examples

[0158] Determine whether the following conversation is likely to be a scam and, if so, raise a red flag.

[0159] Conversation: This is a bank. Please let me know your account number so I can process your request.

[0160] Risk of fraud:

[0161] The system protects users from the risk of fraud and allows them to quickly access family or police assistance.

[0162] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0163] Step 1:

[0164] A user terminal collects the telephone conversation.

[0165] Specific operation: When a user starts a call, voice data is collected in real time using the smartphone's built-in microphone.

[0166] Input: Audio data from a user's phone conversation.

[0167] Output: The audio data is saved to the device.

[0168] Step 2:

[0169] The device converts the collected voice data into text information.

[0170] What it does: The audio data is converted to text using the Google Speech-to-Text API.

[0171] Input: The audio data collected in step 1.

[0172] Output: Textual information in text format.

[0173] Step 3:

[0174] The terminal transmits character information in text format to the server.

[0175] Specific operation: The converted text information is uploaded to a server via the Internet.

[0176] Input: The textual information generated in step 2.

[0177] Output: Textual information stored on the server.

[0178] Step 4:

[0179] The server analyzes the text information using a generative artificial intelligence model.

[0180] Specific operation: A generative artificial intelligence model (such as GPT-4) is launched on the server side using AWS Lambda and analyzes text information.

[0181] Input: The textual information sent to the server in step 3.

[0182] Output: Fraud probability assessment results.

[0183] Step 5:

[0184] The server detects fraud patterns.

[0185] How it works: Generative AI models detect phrases and patterns characteristic of fraud.

[0186] Input: The text parsed in step 4.

[0187] Output: Judgment result on whether or not there is suspicion of fraud.

[0188] Step 6:

[0189] If a fraud pattern is detected, the server sends an emergency notification.

[0190] What it does: Uses AWS SNS to send emergency notifications to pre-registered family members and police agencies.

[0191] Input: The probability of fraud determined in step 5.

[0192] Output: Urgent notification message (e.g. "Potential fraudulent conversation detected").

[0193] Step 7:

[0194] Family members and police are notified and begin responding.

[0195] Specific actions: Family members or police who receive the notification will contact the user and take action.

[0196] Input: The emergency notification sent in step 6.

[0197] Output: Ensuring user safety and implementing fraud prevention measures.

[0198] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0199] 1. Collection of audio data

[0200] The device constantly collects surrounding audio data using a microphone. When a user is having a phone call or a face-to-face conversation, the audio data is monitored in real time. This data collection is equipped with a noise filtering function to ensure clear audio data.

[0201] 2. Transcription of audio data

[0202] The device runs speech recognition software to convert the collected voice data into text. The software analyzes the speech and generates corresponding text data. For example, if a user says, "Is this from my bank?", that phrase is recorded verbatim as text.

[0203] 3. Analysis by Emotion Engine

[0204] The device uses an emotion engine to further analyze the converted voice data. The emotion engine analyzes the tone, speed, and phrasing of the user's voice to identify the user's emotional state. For example, if the user says "Are you sure it's safe?" in an anxious voice, the emotion engine will detect anxiety and fear.

[0205] 4. Fraud Pattern Detection

[0206] The server receives text sent from the device and uses generative artificial intelligence to detect fraud-related patterns. The model compares it with historical fraud datasets to identify phrases and behavioral patterns characteristic of fraud. If a user engages in a conversation that includes the phrase "Please tell me my account information," it is determined to be a potential fraud.

[0207] 5. Integrated judgment of emotional state and deception pattern

[0208] The server combines the analysis results from the emotion engine and the fraud pattern detection results from the generative AI to make a comprehensive judgment. If the user's emotional state is biased toward anxiety or fear, and if fraud-related patterns are detected, the server will assess the possibility of fraud as high.

[0209] 6. Building and sending alerts

[0210] If a fraudulent conversation is deemed likely, the server creates a warning message. This message includes the detected fraudulent pattern, the user's emotional state, the date and time of the detection, and the user's information. The warning message is sent to pre-registered family members and police agencies. For example, a message stating "A potentially fraudulent conversation has been detected and the user is in a state of anxiety" is sent.

[0211] 7. Logging

[0212] The server records detailed logs of all operations performed, including timestamps of operations, emotion engine analysis results, fraud pattern detection results, and details of any alert messages sent, for later analysis and troubleshooting.

[0213] Specific examples

[0214] User Status

[0215] The user is an elderly person who regularly communicates with his family and bank over the phone. One day, he received a call saying, "Hello, is this your bank? I'd like to confirm my account information." During the conversation, the user asked in an anxious voice, "Is this really safe?"

[0216] Device behavior

[0217] The device that receives this conversation collects the voice data and converts it into text information. At the same time, the emotion engine analyzes the tone and speed of the user's voice to detect feelings of anxiety.

[0218] Server Operation

[0219] The user's conversational content and emotional state are sent to the server, which uses generative artificial intelligence to detect fraud-related patterns. For example, if the phrase "I want to check my account information" is detected, and the emotion engine indicates that the user is in an anxious state, it will determine that there is a high possibility of fraud.

[0220] Sending an alert

[0221] If it is determined that there is a high possibility of fraud, the server will send a warning message to the family and the police stating that "a potentially fraudulent conversation has been detected and the user is in a state of anxiety."

[0222] The above is a concrete example of how to implement the present invention by combining an emotion engine. This system protects users from the risk of fraud with a higher degree of accuracy and enables quicker response by family members and the police.

[0223] The processing flow will be explained below.

[0224] Specific steps for collecting and analyzing audio data

[0225] Step 1: Collecting audio data

[0226] The device uses a built-in microphone to capture the user's surrounding sounds in real time, recording all conversations whether the user is making a phone call or talking face-to-face. The device also has a filtering function to reduce environmental noise, ensuring clear audio data is collected.

[0227] Step 2: Temporarily save the audio data

[0228] The device temporarily stores the collected voice data in its internal memory. This ensures stable storage of the voice data and makes it available for subsequent processing. Because the voice data is temporarily stored as a buffer, processing can proceed without interruption.

[0229] Step 3: Transcribe the audio data

[0230] The device uses voice recognition software to analyze the temporarily stored voice data and convert it into text data. For example, a user might say, "Hello, is this from my bank?" and the transcription is then adjusted for accuracy.

[0231] Step 4: Send text information

[0232] The device sends the converted text data and associated metadata (such as user ID and timestamp) to the server, which then promptly transmits the data over the internet connection.

[0233] Step 5: Analyze emotional state

[0234] The server analyzes the voice data received from the device using an emotion engine to evaluate the user's emotional state. The emotion engine analyzes the tone, speed, and strength of the voice to detect emotions such as "anxiety" and "fear." For example, if a user says "Is it really safe?" in a tense voice, the emotion of anxiety is detected.

[0235] Step 6: Detect fraud patterns

[0236] The server then inputs the received text data, along with the emotional state, into a generative AI model to detect fraud-related patterns. The AI ​​model then matches it with a large amount of historical data to identify phrases and phrasing associated with fraudulent activity. For example, the phrase "Please tell me your account information" might be detected.

[0237] Step 7: Comprehensive Decision

[0238] The server integrates the analysis results of the emotion engine and the fraud pattern detection results to comprehensively evaluate the risk of fraud. If the user's emotional state is evaluated as "anxiety" or "fear" and also matches the characteristic fraud patterns, it is determined that there is a very high possibility of fraud.

[0239] Step 8: Building the Alert

[0240] If a fraudulent conversation is detected, the server immediately creates an alert message, which includes the details of the detected fraud, the user's emotional state, the date and time, the user ID, etc. For example, the alert might say, "A potentially fraudulent conversation has been detected and the user is in a state of anxiety."

[0241] Step 9: Sending an alert

[0242] The server then sends the constructed warning message to pre-registered family members and police agencies via email and SMS, ensuring that recipients receive the warning quickly.

[0243] Step 10: Logging

[0244] The server records detailed logs of all operations performed, including timestamps of operations, emotion engine and generative AI results, and details of warning messages. This log data can be used for future analysis and troubleshooting.

[0245] Specific examples

[0246] User Status

[0247] The user is an elderly person who regularly has phone and face-to-face conversations. One day, he received a call saying, "Hello, is this your bank? I'd like to confirm my account information." During the conversation, the user said in an anxious voice, "Is this really safe?"

[0248] Device behavior

[0249] The device that captures this conversation collects the voice data, transcribes it, and immediately sends the results to the server. At the same time, the emotion engine detects anxiety from the user's tone of voice.

[0250] Server Operation

[0251] Based on the received text data and the user's emotional state, the server uses generative artificial intelligence to detect fraud patterns. For example, if the phrase "I want to check my account information" matches the characteristics of a fraud, and the user's anxiety is detected, the server determines that there is a high possibility of fraud.

[0252] Sending an alert

[0253] Because it is determined that there is a high possibility of fraud, the server sends a warning message to the family and the police stating that "a potentially fraudulent conversation has been detected and the user is in a state of anxiety."

[0254] This provides users with far greater protection from the risk of fraud and allows for faster response.

[0255] Example 2

[0256] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0257] In modern society, voice-mediated fraud is on the rise, and elderly people and those who are not familiar with information are particularly vulnerable to becoming victims. Conventional fraud detection systems have technologies to recognize fraudulent patterns, but they have limitations in their methods of analyzing the user's emotional state in real time and making an integrated judgment to further increase the likelihood of fraud. This has led to frequent false positives, increasing the possibility that appropriate warnings will not be issued in situations where protection is truly needed.

[0258] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text information, generative artificial intelligence means for detecting fraud-related patterns using the converted text information, emotion analysis means for analyzing the voice data to identify the user's emotional state, and means for integrating the analysis results of the emotion analysis means and the fraud-related pattern detection results to make a comprehensive judgment. This enables highly accurate fraud detection that takes the user's emotional state into consideration, and effectively protects users from the risk of fraud.

[0259] "Audio data" is signal information including the user's and surrounding sounds.

[0260] "Text information" is information in text format obtained by analyzing voice data.

[0261] "Generative AI methods" are AI techniques that learn from past data and generate and identify specific patterns based on newly input data.

[0262] "Emotion analysis means" refers to technology and devices that identify the user's emotional state from voice data and text information.

[0263] "Fraud-related patterns" refer to specific phrases or behavioral characteristics that characterize fraudulent activity.

[0264] "Means for making a comprehensive judgment" refers to the technology and process that integrates the results of sentiment analysis and fraud-related pattern detection to make a final judgment.

[0265] "Means for sending an alert" means the functions and techniques for constructing and sending an alert message to a designated recipient when fraud is determined to be likely.

[0266] The "historical fraud dataset" is a database that collects data on fraud cases that have occurred in the past.

[0267] "Pre-registered family members and police agencies" are emergency contacts that the user has preset in the system.

[0268] This system collects and analyzes user voice data to detect and warn of fraudulent activity early. The following hardware and software are used to implement this system:

[0269] 1. Collection of audio data

[0270] The device uses a microphone to constantly collect sounds around the user. Noise-canceling technology is used to ensure clear audio data. Specifically, a smartphone or a dedicated audio collection device is used.

[0271] 2. Transcription of audio data

[0272] The device runs speech recognition software, such as the Google Speech-to-Text API or Microsoft Azure Cognitive Services, to convert the collected voice data into text. For example, if a user says, "Is this from my bank?", the exact phrase is recorded as text data.

[0273] 3. Analysis by Emotion Engine

[0274] The device runs an emotion engine to analyze the voice data converted into text. The emotion engine analyzes the user's tone of voice, speech rate, intonation, etc. to identify the user's emotional state. Emotion engines used include IBM Watson Tone Analyzer and Lexalytics' Sentiment Analysis API. For example, if a user says "Is this really okay?" in an anxious voice, the emotion engine will detect the "anxiety."

[0275] 4. Fraud Pattern Detection

[0276] The server receives the text sent from the device and uses a generative artificial intelligence model to detect fraud-related patterns. This model compares past fraud datasets. Specifically, OpenAI's GPT-3 and Google BERT are used as generative technologies. If a user engages in a conversation that includes a phrase such as "Please tell me my account information," this is likely to be judged as fraud.

[0277] 5. Integrated judgment of emotional state and deception pattern

[0278] The server combines the emotion analysis results from the emotion engine with the fraud pattern detection results from the generative AI to make a comprehensive judgment. For example, if the emotion analysis result is "anxiety" and the fraud pattern matches, it will assess the possibility of fraud. This information is combined according to a pre-set algorithm by the system.

[0279] 6. Building and sending alerts

[0280] If it is determined that there is a high possibility of fraud, the server will construct an alert message. This message will include the detected fraud pattern, the user's emotional state, the detection date and time, and user information. The alert message will be sent to pre-registered family members and police agencies. In a specific implementation, the alert email will be sent using the SMTP protocol.

[0281] (Example)

[0282] The user is an elderly person who regularly communicates with family and the bank over the phone. One day, he receives a call saying, "Hello, is this the bank? I'd like to check my account information." During the conversation, the user says in an anxious voice, "Is it really safe?" The device that receives this conversation collects the voice data and converts it into text. At the same time, the emotion engine analyzes the tone and speed of the user's voice to detect feelings of anxiety.

[0283] The server receives the user's conversation content and emotional state and uses generative artificial intelligence to detect fraud-related patterns. For example, the phrase "I want to check my account information" is detected, and the emotion engine indicates that the user is in an anxious state. As a result, it is determined that there is a high possibility of fraud, and the server sends an alert message to the user's family and the police stating, "A conversation that may be fraudulent has been detected, and the user is in an anxious state."

[0284] The above is a concrete example of how to implement a fraud detection system that combines an emotion engine. This system will protect users from fraud risks with a higher degree of accuracy and enable quicker response by family members and the police.

[0285] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0286] Step 1:

[0287] The device uses a microphone to collect audio data from the user's surroundings. It then uses the device's built-in noise-canceling technology to capture clear audio data, reducing noise and providing highly accurate audio data. The collected audio data is temporarily stored in the device's memory.

[0288] Input: Ambient audio

[0289] Output: Clear audio data

[0290] Step 2:

[0291] The device sends the collected voice data to the Google Speech-to-Text API and converts it into text. During this process, the voice data is analyzed and corresponding text data is generated. The generated text data is then passed on to the next processing step.

[0292] Input: Clear voice data

[0293] Output: Character information (text data)

[0294] Step 3:

[0295] The device sends the text information to the emotion engine, which uses IBM Watson Tone Analyzer. This engine analyzes the tone, speed, and intonation of the words in the text data to identify the user's emotional state. The analysis results in an emotion label, such as "anxiety" or "joy."

[0296] Input: Character information (text data)

[0297] Output: Sentiment analysis results (emotion labels)

[0298] Step 4:

[0299] The device sends the text information and the results of the emotion analysis to the server. The server inputs the text information into a generative AI model as a prompt sentence. The prompt sentence is a specific sentence based on the text information. The generative AI model (e.g., GPT-3) analyzes the prompt sentence and detects fraud-related patterns. The detection result is a label such as "Fraud pattern detected" or "No problem."

[0300] Input: Text information, emotion analysis results

[0301] Output: Fraud-related pattern detection results

[0302] Step 5:

[0303] The server integrates the results of the sentiment analysis and the fraud-related pattern detection to make a comprehensive judgment. If the sentiment analysis result is "anxiety" and a fraud pattern is detected, the possibility of fraud is assessed as high. The result of this comprehensive judgment is used to construct a warning message.

[0304] Input: Sentiment analysis results, fraud-related pattern detection results

[0305] Output: Overall judgment result

[0306] Step 6:

[0307] The server then creates a warning message based on the comprehensive judgment results, which includes the detected fraud pattern, the user's emotional state, the detection date and time, user information, etc. This warning message is then sent to pre-registered family members and police agencies using the SMTP protocol.

[0308] Input: Overall judgment result

[0309] Output: Warning message

[0310] Step 7:

[0311] The server records detailed logs of all processing steps, including processing timestamps, sentiment analysis results, fraud pattern detection results, and details of any alert messages sent. This log is stored for later analysis and troubleshooting.

[0312] Input: All processing results

[0313] Output: Log data

[0314] (Application example 2)

[0315] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0316] The risk of elderly and socially vulnerable people falling victim to fraud continues to increase. Conventional fraud prevention systems have limitations in their ability to detect fraudulent patterns and do not adequately analyze the user's emotional state, making them unable to completely prevent fraud. There is a need for a system that can resolve these issues, more accurately and quickly identify potential fraud, and send alerts.

[0317] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0318] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text information, means for analyzing the user's emotional state based on the converted text information, generative artificial intelligence means for detecting fraud-related patterns using the analysis results, and means for sending an alert when a fraud-related pattern is detected. This makes it possible to comprehensively analyze the user's emotional state and fraud-related patterns, and to send an alert quickly and accurately when there is a high possibility of fraud.

[0319] The "means for collecting voice data" refers to a device that has the function of capturing and recording external voice in real time using a microphone on the terminal.

[0320] The "means for converting collected voice data into text information" is a device that has the function of analyzing voice data using voice recognition software and generating corresponding text data.

[0321] The "means for analyzing the user's emotional state based on the converted text information" is a device that has the function of analyzing the user's speaking tone, speed, choice of words, etc., and identifying the user's emotional state using an emotion engine.

[0322] A "generative artificial intelligence means" is an artificial intelligence system that has the ability to learn specific patterns or trends based on past data sets and detect those patterns or trends based on newly input data.

[0323] A "means for detecting fraud-related patterns" is a device capable of using generative artificial intelligence to identify phrases and behavioral patterns characteristic of fraudulent activity.

[0324] The "means for sending an alert" is a device that has the function of automatically sending an alert message to pre-registered recipients when there is a high possibility of fraud.

[0325] The present invention provides a system that collects voice data from a user, analyzes the data to detect the user's emotional state and fraud-related patterns, and sends alerts as needed.

[0326] 1. Collection of audio data

[0327] The device constantly collects voice data using a microphone, allowing it to capture conversations in real time as users interact. The collected voice data is also equipped with a noise filtering function to ensure clear voice data.

[0328] 2. Transcription of audio data

[0329] The collected voice data is converted into text using voice recognition software. For example, Google Speech-to-Text API is commonly used. This software analyzes the voice data and generates corresponding text data. For example, if a user says, "Is this from my bank?", that phrase is accurately recorded as text.

[0330] 3. Analysis by Emotion Engine

[0331] The server then uses an emotion engine to further analyze the converted voice data. The IBM Watson Tone Analyzer is often used as the emotion engine. This engine analyzes the tone, speed, and phrasing of the user's voice to identify the user's emotional state. For example, if the user says "Is it really safe?" in an anxious voice, the emotion engine will detect anxiety.

[0332] 4. Fraud Pattern Detection

[0333] Generative AI is used to detect fraud patterns from the converted text information. Examples of generative AI include OpenAI GPT-3. This method compares with past fraud datasets to identify fraud-related behavioral patterns and phrases. For example, if a conversation includes the phrase "Please tell me your account information," the system will determine that there is a high possibility of fraud.

[0334] 5. Sending an Alert

[0335] If it determines that there is a high possibility of fraud, the server will send a warning message to pre-registered contacts (family or police agencies). This message will include the detected fraud pattern, the user's emotional state, the date and time of detection, and user information. For example, a warning message will be sent stating, "A potentially fraudulent conversation has been detected and the user is in a state of anxiety."

[0336] 6. Logging

[0337] The server records a detailed log of all operations performed, including timestamps of operations, emotion engine analysis results, fraud pattern detection results, details of alert messages sent, etc. This log can be used for later analysis and troubleshooting.

[0338] Specific examples

[0339] For example, suppose an elderly person receives a phone call saying, "Hello, is this the bank? I'd like to check my account information." The device receiving this conversation collects voice data and converts it into text. At the same time, an emotion engine analyzes the tone and speed of the user's voice to detect feelings of anxiety. The content of the user's conversation and emotional state are sent to a server, which uses generative artificial intelligence to detect fraud-related patterns. For example, if the phrase "I'd like to check my account information" is detected and the emotion engine indicates that the user is in an anxious state, it is determined that there is a high possibility of fraud. As a result, the server sends an alert message to the family and police stating, "A potentially fraudulent conversation has been detected and the user is in an anxious state."

[0340] Prompt Sentence Examples

[0341] Determine if the following text contains any signs of fraud: "Hello, is this your bank? I'd like to verify my account information."

[0342] This invention provides users with high accuracy protection from fraud and can send alerts quickly, allowing for a quick response by family members or police.

[0343] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0344] Step 1:

[0345] The device uses a microphone to collect surrounding audio data, which is then saved in real time and filtered to ensure clear audio.

[0346] Input: External audio

[0347] Output: filtered audio data

[0348] Step 2:

[0349] The filtered voice data is converted into text using voice recognition software (e.g., Google Speech-to-Text API).

[0350] Input: filtered audio data

[0351] Output: Character information (text data)

[0352] Step 3:

[0353] The text information is sent to a server, and the user's emotional state is analyzed using an emotion engine (e.g., IBM Watson Tone Analyzer). The emotion engine analyzes the tone, speed, and phrasing of the text data to identify the user's emotion.

[0354] Input: Character information (text data)

[0355] Output: Emotion analysis results (anxiety, anger, joy, etc.)

[0356] Step 4:

[0357] The server uses generative AI (e.g., OpenAI GPT-3) to detect fraud-related patterns based on the sentiment analysis results and text information. This AI compares the results with fraud patterns learned from past datasets to find signs of fraud.

[0358] Input: Emotion analysis results, character information (text data)

[0359] Output: Fraud pattern detection results

[0360] Step 5:

[0361] If a fraudulent activity is detected, the server sends a warning message to pre-registered contacts, which includes information about the detected fraudulent activity, the user's emotional state, the date and time of the detection, and other user information.

[0362] Input: Fraud pattern detection results, sentiment analysis results

[0363] Output: Warning message

[0364] Step 6:

[0365] The server records detailed logs of all operations performed, including timestamps of operations, emotion engine analysis results, fraud pattern detection results, and details of any alert messages sent, for later analysis and troubleshooting.

[0366] Input: Output results of each step

[0367] Output: Detailed log information

[0368] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0369] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0370] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0371] [Second embodiment]

[0372] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0373] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0374] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0375] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0376] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0377] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0378] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0379] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0380] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0381] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0382] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0383] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0384] 1. Collection of audio data

[0385] The device constantly monitors surrounding voice data using a microphone. The device also has a voice recognition sensor that collects user conversations in real time. For example, voice data is acquired when the user is talking on the phone or talking face-to-face with family members.

[0386] 2. Transcription of audio data

[0387] The device uses software to convert the collected voice data into text. This software uses speech recognition technology to analyze the voice data and generate corresponding text. For example, if a user says, "Is this from my bank?", this phrase will be recorded as text.

[0388] 3. Fraud Pattern Detection

[0389] The server receives text sent from the device and uses generative artificial intelligence to detect fraud-related patterns. This AI model is trained on historical fraud datasets and determines in real time whether the user's conversation matches the characteristics of fraud. For example, if a phrase like "Can you tell me my account information?" is detected in the conversation, it will mark it as a possible fraud.

[0390] 4. Sending alerts

[0391] If a fraud-related pattern is detected, the server immediately sends an alert. Recipients of the alert include pre-registered family members and police agencies. For example, if a user utters a typical fraudulent phrase like "Do I need to provide my account number?", the server will send an alert message to family members and the police saying, "A possible fraudulent conversation has been detected."

[0392] Specific examples

[0393] User Status

[0394] The user is an elderly person who regularly communicates with his family and bank over the phone. One day, he receives a call saying, "Hello, is this the bank? I'd like to check my account information."

[0395] Device behavior

[0396] The terminal that receives this conversation immediately collects the voice data and converts it into text information.

[0397] Server Operation

[0398] The user's conversation is sent to a server, which uses generative artificial intelligence to detect fraud patterns. In this case, the phrase "I would like to check my account information" matches the fraud pattern, so it is immediately determined to be a potential fraud.

[0399] Sending alerts

[0400] If a fraudulent conversation is detected, the server sends a warning message to the user's family and the police, saying, "A potential fraudulent conversation has been detected." This allows the family and the police to respond quickly and prevent the user from becoming a victim of fraud.

[0401] The above is a specific embodiment for carrying out the present invention. The present invention protects users from the risk of fraud and allows them to quickly receive support from their families or the police.

[0402] The processing flow will be explained below.

[0403] Specific steps for collecting and processing audio data

[0404] Step 1: Collecting audio data

[0405] The device activates voice recognition software and captures surrounding audio through a microphone, monitoring all user conversations in real time while filtering out background noise and other distracting sounds.

[0406] Step 2: Temporarily save the audio data

[0407] The device temporarily stores the collected voice data in its internal memory. This data is temporarily held for subsequent processing. As voice data is collected, it is accumulated in a buffer.

[0408] Step 3: Transcribe the audio data

[0409] The device uses voice recognition software to convert the temporarily stored voice data into text. The text generated by this process is a textual representation of what the user said. For example, "Hello, is this from my bank?"

[0410] Step 4: Send text information

[0411] The device then sends the converted text data, including necessary metadata (such as a timestamp and user ID), to a server via the Internet.

[0412] Step 5: Detect fraud patterns

[0413] The server analyzes the received text and uses generative artificial intelligence to detect fraud patterns. Specifically, it compares it with historical data sets and identifies phrases and wording characteristic of fraud. This analysis is done in real time.

[0414] Step 6: Determine the likelihood of fraud

[0415] The server evaluates the likelihood of fraud based on the results of the generative AI. If the algorithm determines that there is a high risk of fraud, the result is recorded.

[0416] Step 7: Building the Alert

[0417] If the server detects possible fraud, it constructs an alert message that includes the nature of the detected fraud, the date and time, and the affected user.

[0418] Step 8: Sending an alert

[0419] The server sends alert messages to pre-registered family members and police agencies via email and SMS, allowing recipients to receive the alerts immediately and respond quickly.

[0420] Step 9: Logging

[0421] The server records detailed logs of all operations it performs, including timestamps of operations, fraud patterns detected, and details of warning messages sent, for later analysis and troubleshooting.

[0422] Example 1

[0423] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0424] Today, the risk of becoming a victim of fraud is increasing, especially among the elderly, and there is a need to detect signs of fraud in real time and respond quickly. Furthermore, as fraud methods become more sophisticated, it is difficult to respond with traditional manual monitoring and vigilance, and cooperation from family members and police agencies is essential. Therefore, it is necessary to realize a fraud detection system that uses voice data.

[0425] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0426] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text information, means for transmitting the converted text information to the server, generative artificial intelligence means for detecting fraud-related patterns using the text information received by the server, and means for sending an alert when a fraud-related pattern is detected. This makes it possible to detect signs of fraud in real time, send an alert quickly, and protect users from fraud.

[0427] "Voice data" refers to information that is a digital recording of the user's speech and surrounding sounds.

[0428] "Collection means" refers to a device or method for acquiring audio data using a microphone, sensor, etc.

[0429] "Text information" refers to text data converted from collected voice data using voice recognition technology.

[0430] "Means of converting into textual information" refers to the process of converting audio data into textual data using speech recognition software or algorithms.

[0431] The "means for transmitting the converted character information" is a mechanism for transferring the generated text data to a server via a secure protocol.

[0432] "Generative artificial intelligence" is an AI technology that learns from past data sets and makes predictions and judgments in real time about new data.

[0433] "Fraud-related patterns" refer to specific phrases or patterns of behavior that indicate fraudulent activity.

[0434] "Means for sending an alert" refers to a method for sending a warning message to pre-registered recipients when a fraud-related pattern is detected.

[0435] A "historical fraud dataset" is a collection of previously recorded fraud cases and patterns.

[0436] "Family and Law Enforcement" refers to contacts that include the user's pre-registered relatives and law enforcement agencies.

[0437] MODE FOR CARRYING OUT THE INVENTION

[0438] The present invention relates to a system that uses specific hardware and software to provide real-time fraud protection to users, and a specific embodiment of the system is described below.

[0439] Audio data collection

[0440] The device constantly collects surrounding audio data using a microphone. The device is equipped with a high-performance voice recognition sensor that captures the user's conversation in real time. The device can be a smartphone, tablet, smart speaker, or other device. For example, when a user is talking on the phone, the device can be equipped with a function to record the conversation.

[0441] Transcription of audio data

[0442] The device uses speech recognition software to convert collected voice data into text. Specifically, it uses speech recognition technologies such as the Google Speech-to-Text API and Microsoft Azure Cognitive Services Speech API. This software analyzes the voice data and generates corresponding text. For example, if a user says, "Is this from my bank?", this phrase will be recorded as text on the device.

[0443] Sending audio data

[0444] The device sends the generated text data to the server in an encrypted format using a secure protocol (e.g., HTTPS), which ensures data security and privacy.

[0445] Fraud pattern detection

[0446] The server receives the text sent from the device and begins analyzing it in real time using generative artificial intelligence (e.g., OpenAI GPT). The software on the server is trained on historical fraud datasets and determines whether the text matches the characteristics of fraud. For example, if the phrase "Please tell me your account information" is detected, it will be marked as a possible fraud.

[0447] Sending alerts

[0448] If a fraud-related pattern is detected, the server immediately sends an alert. Recipients of the alert include pre-registered family members and police agencies. For example, if a user says, "Do I need to provide my account number?", the server will send an alert message to family members and the police saying, "A potentially fraudulent conversation has been detected."

[0449] Specific examples

[0450] User Status

[0451] The user is an elderly person who regularly communicates with family and financial institutions over the phone. One day, he receives a call saying, "Hello, is this the bank? I'd like to check my account information."

[0452] Device behavior

[0453] The device that receives this conversation immediately collects the voice data and converts it into text information. The device is a smartphone, and the voice data is converted into text using the Google Speech-to-Text API.

[0454] Server Operation

[0455] The user's conversation is sent to a server, which uses generative artificial intelligence to detect fraud patterns. In this case, the phrase "I would like to check my account information" matches the fraud pattern, so it is immediately determined to be a potential fraud.

[0456] Sending alerts

[0457] If a fraudulent conversation is detected, the server sends a warning message to the user's family and the police, saying, "A potential fraudulent conversation has been detected." This allows the family and the police to respond quickly and prevent the user from becoming a victim of fraud.

[0458] Prompt Sentence Examples

[0459] "If a user calls and says, 'Is this your bank? I'd like to confirm my account information,' we'll mark it as a fraud pattern and send an alert message to their family and the police."

[0460] This explains the configuration of a system for specifically implementing the present invention, which protects users from the risk of fraud and sends prompt alerts to obtain cooperation from family members and the police.

[0461] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0462] Step 1:

[0463] The device constantly collects surrounding audio data using a microphone. When the user speaks, the device captures the audio and records it in digital form. The input is the user's speech, and the output is digital audio data. For example, a user might record themselves saying, "Hello, is this the bank?"

[0464] Step 2:

[0465] The device converts the collected voice data into text using the Google Speech-to-Text API. During this conversion process, voice recognition technology is used to analyze the voice data and generate corresponding text. The input is digital voice data, and the output is text data. Specifically, the voice saying "Hello, is this the bank?" is converted into text "Hello, is this the bank?"

[0466] Step 3:

[0467] The terminal securely transmits the generated text data to the server using the HTTPS protocol. During the data transmission process, data security and privacy are guaranteed by encryption. The input is text data, and the output is secure data transmission to the server. Specifically, the text data "Hello, is this the bank?" is transmitted to the server.

[0468] Step 4:

[0469] The server analyzes the received text data in real time. It runs a generative artificial intelligence (e.g., OpenAI GPT) and compares it with past fraud datasets to detect fraud-related patterns. The input is text data, and the output is the detection result of fraud-related patterns. For example, if the phrase "Please tell me your account information" is detected, it is marked as a possible fraud.

[0470] Step 5:

[0471] If a fraud-related pattern is detected, the server immediately generates an alert and sends it to pre-registered family members and police agencies. During this process, the generated alert message is forwarded to each contact in the distribution list. The input is a fraud-related text phrase, and the output is the alert message sent to family members and police agencies. Specifically, the alert message sent is "A potential fraudulent conversation has been detected."

[0472] Through these steps, the system protects users from the risk of fraud and can quickly issue alerts to prompt action from relevant authorities.

[0473] (Application example 1)

[0474] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0475] In modern society, cases of elderly people falling victim to telephone fraud are increasing. To prevent such fraud, real-time fraud detection and rapid response methods are required. However, conventional methods are insufficient to efficiently monitor telephone conversations and detect possible fraud, making it difficult to completely prevent fraud. Technology to address this issue is needed.

[0476] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0477] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text information, generative artificial intelligence means for detecting fraud-related patterns using the converted text information, means for sending an alert when a fraud-related pattern is detected, means for converting voice data into text using voice recognition software, means for analyzing fraud patterns using a generative artificial intelligence model using a cloud service, and notification means for sending an emergency notification when fraud is detected. This makes it possible to detect potentially fraudulent telephone conversations in real time and quickly notify family members and the police.

[0478] "Audio data" refers to information that records audio in digital format.

[0479] "Means of collection" refers to devices and software for acquiring and storing voice data.

[0480] "Means for converting into textual information" refers to technology or software that converts audio data into text format.

[0481] "Generative artificial intelligence" refers to an AI model that learns from past data and analyzes new data.

[0482] "Detecting fraud-related patterns" is the process of determining whether certain phrases or behaviors are associated with fraud.

[0483] An "alert means" is a system or technology for providing notification in the event of suspected fraud.

[0484] "Speech recognition software" is a program that converts speech into text in real time.

[0485] "Cloud services" are computer resources and data analysis services provided via the Internet.

[0486] A "generative artificial intelligence model" is an AI model used to detect fraud patterns from text data.

[0487] A "notification mechanism" is a system capable of sending emergency notifications based on detected fraud patterns.

[0488] This invention is a real-time fraud detection and emergency notification system that monitors users' phone conversations using their smartphones and can take immediate action if fraudulent activity is suspected.

[0489] System configuration

[0490] The system consists of the following main components:

[0491] 1. User device (smartphone)

[0492] 2. Speech recognition software (Google Speech-to-Text API)

[0493] 3. Cloud Services (Amazon Web Services: AWS Lambda and AWS SNS)

[0494] 4. Generative AI models (e.g., GPT-4)

[0495] Hardware and Software Use

[0496] The user device is equipped with a built-in microphone that is used to collect real-time phone conversations, and this voice data is converted into text using the Google Speech-to-Text API.

[0497] The converted text is sent to AWS Lambda, a cloud service, and analyzed by a generative AI model (e.g., GPT-4), which is trained on historical fraud datasets to detect phrases and patterns characteristic of fraud.

[0498] If a suspicious pattern is detected, AWS SNS will send an emergency notification to pre-registered family members and law enforcement agencies, including a message such as "A potential fraudulent conversation has been detected."

[0499] Specific examples of processing

[0500] User Status

[0501] The user is an elderly person who regularly communicates with family, banks, etc. by telephone. One day, he receives a call saying, "Hello, this is the bank. I would like to confirm my account information."

[0502] Device behavior

[0503] The user's device collects this conversation in real time and converts the audio data into text using the Google Speech-to-Text API.

[0504] Server Operation

[0505] The converted text is sent to AWS Lambda in the cloud and analyzed using a generative artificial intelligence model (e.g., GPT-4). If the detected patterns match the characteristics of a fraud, an emergency notification is sent to family members and police via AWS SNS.

[0506] Prompt Sentence Examples

[0507] Determine whether the following conversation is likely to be a scam and, if so, raise a red flag.

[0508] Conversation: This is a bank. Please let me know your account number so I can process your request.

[0509] Risk of fraud:

[0510] The system protects users from the risk of fraud and allows them to quickly access family or police assistance.

[0511] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0512] Step 1:

[0513] A user terminal collects the telephone conversation.

[0514] Specific operation: When a user starts a call, voice data is collected in real time using the smartphone's built-in microphone.

[0515] Input: Audio data from a user's phone conversation.

[0516] Output: The audio data is saved to the device.

[0517] Step 2:

[0518] The device converts the collected voice data into text information.

[0519] What it does: The audio data is converted to text using the Google Speech-to-Text API.

[0520] Input: The audio data collected in step 1.

[0521] Output: Textual information in text format.

[0522] Step 3:

[0523] The terminal transmits character information in text format to the server.

[0524] Specific operation: The converted text information is uploaded to a server via the Internet.

[0525] Input: The textual information generated in step 2.

[0526] Output: Textual information stored on the server.

[0527] Step 4:

[0528] The server analyzes the text information using a generative artificial intelligence model.

[0529] Specific operation: A generative artificial intelligence model (such as GPT-4) is launched on the server side using AWS Lambda and analyzes text information.

[0530] Input: The textual information sent to the server in step 3.

[0531] Output: Fraud probability assessment results.

[0532] Step 5:

[0533] The server detects fraud patterns.

[0534] How it works: Generative AI models detect phrases and patterns characteristic of fraud.

[0535] Input: The text parsed in step 4.

[0536] Output: Judgment result on whether or not there is suspicion of fraud.

[0537] Step 6:

[0538] If a fraud pattern is detected, the server sends an emergency notification.

[0539] What it does: Uses AWS SNS to send emergency notifications to pre-registered family members and police agencies.

[0540] Input: The probability of fraud determined in step 5.

[0541] Output: Urgent notification message (e.g. "Potential fraudulent conversation detected").

[0542] Step 7:

[0543] Family members and police are notified and begin responding.

[0544] Specific actions: Family members or police who receive the notification will contact the user and take action.

[0545] Input: The emergency notification sent in step 6.

[0546] Output: Ensuring user safety and implementing fraud prevention measures.

[0547] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0548] 1. Collection of audio data

[0549] The device constantly collects surrounding audio data using a microphone. When a user is having a phone call or a face-to-face conversation, the audio data is monitored in real time. This data collection is equipped with a noise filtering function to ensure clear audio data.

[0550] 2. Transcription of audio data

[0551] The device runs speech recognition software to convert the collected voice data into text. The software analyzes the speech and generates corresponding text data. For example, if a user says, "Is this from my bank?", that phrase is recorded verbatim as text.

[0552] 3. Analysis by Emotion Engine

[0553] The device uses an emotion engine to further analyze the converted voice data. The emotion engine analyzes the tone, speed, and phrasing of the user's voice to identify the user's emotional state. For example, if the user says "Are you sure it's safe?" in an anxious voice, the emotion engine will detect anxiety and fear.

[0554] 4. Fraud Pattern Detection

[0555] The server receives text sent from the device and uses generative artificial intelligence to detect fraud-related patterns. The model compares it with historical fraud datasets to identify phrases and behavioral patterns characteristic of fraud. If a user engages in a conversation that includes the phrase "Please tell me my account information," it is determined to be a potential fraud.

[0556] 5. Integrated judgment of emotional state and deception pattern

[0557] The server combines the analysis results from the emotion engine and the fraud pattern detection results from the generative AI to make a comprehensive judgment. If the user's emotional state is biased toward anxiety or fear, and if fraud-related patterns are detected, the server will assess the possibility of fraud as high.

[0558] 6. Building and sending alerts

[0559] If a fraudulent conversation is deemed likely, the server creates a warning message. This message includes the detected fraudulent pattern, the user's emotional state, the date and time of the detection, and the user's information. The warning message is sent to pre-registered family members and police agencies. For example, a message stating "A potentially fraudulent conversation has been detected and the user is in a state of anxiety" is sent.

[0560] 7. Logging

[0561] The server records detailed logs of all operations performed, including timestamps of operations, emotion engine analysis results, fraud pattern detection results, and details of any alert messages sent, for later analysis and troubleshooting.

[0562] Specific examples

[0563] User Status

[0564] The user is an elderly person who regularly communicates with his family and bank over the phone. One day, he received a call saying, "Hello, is this your bank? I'd like to confirm my account information." During the conversation, the user asked in an anxious voice, "Is this really safe?"

[0565] Device behavior

[0566] The device that receives this conversation collects the voice data and converts it into text information. At the same time, the emotion engine analyzes the tone and speed of the user's voice to detect feelings of anxiety.

[0567] Server Operation

[0568] The user's conversational content and emotional state are sent to the server, which uses generative artificial intelligence to detect fraud-related patterns. For example, if the phrase "I want to check my account information" is detected, and the emotion engine indicates that the user is in an anxious state, it will determine that there is a high possibility of fraud.

[0569] Sending an alert

[0570] If it is determined that there is a high possibility of fraud, the server will send a warning message to the family and the police stating that "a potentially fraudulent conversation has been detected and the user is in a state of anxiety."

[0571] The above is a concrete example of how to implement the present invention by combining an emotion engine. This system protects users from the risk of fraud with a higher degree of accuracy and enables quicker response by family members and the police.

[0572] The processing flow will be explained below.

[0573] Specific steps for collecting and analyzing audio data

[0574] Step 1: Collecting audio data

[0575] The device uses a built-in microphone to capture the user's surrounding sounds in real time, recording all conversations whether the user is making a phone call or talking face-to-face. The device also has a filtering function to reduce environmental noise, ensuring clear audio data is collected.

[0576] Step 2: Temporarily save the audio data

[0577] The device temporarily stores the collected voice data in its internal memory. This ensures stable storage of the voice data and makes it available for subsequent processing. Because the voice data is temporarily stored as a buffer, processing can proceed without interruption.

[0578] Step 3: Transcribe the audio data

[0579] The device uses voice recognition software to analyze the temporarily stored voice data and convert it into text data. For example, a user might say, "Hello, is this from my bank?" and the transcription is then adjusted for accuracy.

[0580] Step 4: Send text information

[0581] The device sends the converted text data and associated metadata (such as user ID and timestamp) to the server, which then promptly transmits the data over the internet connection.

[0582] Step 5: Analyze emotional state

[0583] The server analyzes the voice data received from the device using an emotion engine to evaluate the user's emotional state. The emotion engine analyzes the tone, speed, and strength of the voice to detect emotions such as "anxiety" and "fear." For example, if a user says "Is it really safe?" in a tense voice, the emotion of anxiety is detected.

[0584] Step 6: Detect fraud patterns

[0585] The server then inputs the received text data, along with the emotional state, into a generative AI model to detect fraud-related patterns. The AI ​​model then matches it with a large amount of historical data to identify phrases and phrasing associated with fraudulent activity. For example, the phrase "Please tell me your account information" might be detected.

[0586] Step 7: Comprehensive Decision

[0587] The server integrates the analysis results of the emotion engine and the fraud pattern detection results to comprehensively evaluate the risk of fraud. If the user's emotional state is evaluated as "anxiety" or "fear" and also matches the characteristic fraud patterns, it is determined that there is a very high possibility of fraud.

[0588] Step 8: Building the Alert

[0589] If a fraudulent conversation is detected, the server immediately creates an alert message, which includes the details of the detected fraud, the user's emotional state, the date and time, the user ID, etc. For example, the alert might say, "A potentially fraudulent conversation has been detected and the user is in a state of anxiety."

[0590] Step 9: Sending an alert

[0591] The server then sends the constructed warning message to pre-registered family members and police agencies via email and SMS, ensuring that recipients receive the warning quickly.

[0592] Step 10: Logging

[0593] The server records detailed logs of all operations performed, including timestamps of operations, emotion engine and generative AI results, and details of warning messages. This log data can be used for future analysis and troubleshooting.

[0594] Specific examples

[0595] User Status

[0596] The user is an elderly person who regularly has phone and face-to-face conversations. One day, he received a call saying, "Hello, is this your bank? I'd like to confirm my account information." During the conversation, the user said in an anxious voice, "Is this really safe?"

[0597] Device behavior

[0598] The device that captures this conversation collects the voice data, transcribes it, and immediately sends the results to the server. At the same time, the emotion engine detects anxiety from the user's tone of voice.

[0599] Server Operation

[0600] Based on the received text data and the user's emotional state, the server uses generative artificial intelligence to detect fraud patterns. For example, if the phrase "I want to check my account information" matches the characteristics of a fraud, and the user's anxiety is detected, the server determines that there is a high possibility of fraud.

[0601] Sending an alert

[0602] Because it is determined that there is a high possibility of fraud, the server sends a warning message to the family and the police stating that "a potentially fraudulent conversation has been detected and the user is in a state of anxiety."

[0603] This provides users with far greater protection from the risk of fraud and allows for faster response.

[0604] Example 2

[0605] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0606] In modern society, voice-mediated fraud is on the rise, and elderly people and those who are not familiar with information are particularly vulnerable to becoming victims. Conventional fraud detection systems have technologies to recognize fraudulent patterns, but they have limitations in their methods of analyzing the user's emotional state in real time and making an integrated judgment to further increase the likelihood of fraud. This has led to frequent false positives, increasing the possibility that appropriate warnings will not be issued in situations where protection is truly needed.

[0607] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text information, generative artificial intelligence means for detecting fraud-related patterns using the converted text information, emotion analysis means for analyzing the voice data to identify the user's emotional state, and means for integrating the analysis results of the emotion analysis means and the fraud-related pattern detection results to make a comprehensive judgment. This enables highly accurate fraud detection that takes the user's emotional state into consideration, and effectively protects users from the risk of fraud.

[0608] "Audio data" is signal information including the user's and surrounding sounds.

[0609] "Text information" is information in text format obtained by analyzing voice data.

[0610] "Generative AI methods" are AI techniques that learn from past data and generate and identify specific patterns based on newly input data.

[0611] "Emotion analysis means" refers to technology and devices that identify the user's emotional state from voice data and text information.

[0612] "Fraud-related patterns" refer to specific phrases or behavioral characteristics that characterize fraudulent activity.

[0613] "Means for making a comprehensive judgment" refers to the technology and process that integrates the results of sentiment analysis and fraud-related pattern detection to make a final judgment.

[0614] "Means for sending an alert" means the functions and techniques for constructing and sending an alert message to a designated recipient when fraud is determined to be likely.

[0615] The "historical fraud dataset" is a database that collects data on fraud cases that have occurred in the past.

[0616] "Pre-registered family members and police agencies" are emergency contacts that the user has preset in the system.

[0617] This system collects and analyzes user voice data to detect and warn of fraudulent activity early. The following hardware and software are used to implement this system:

[0618] 1. Collection of audio data

[0619] The device uses a microphone to constantly collect sounds around the user. Noise-canceling technology is used to ensure clear audio data. Specifically, a smartphone or a dedicated audio collection device is used.

[0620] 2. Transcription of audio data

[0621] The device runs speech recognition software, such as the Google Speech-to-Text API or Microsoft Azure Cognitive Services, to convert the collected voice data into text. For example, if a user says, "Is this from my bank?", the exact phrase is recorded as text data.

[0622] 3. Analysis by Emotion Engine

[0623] The device runs an emotion engine to analyze the voice data converted into text. The emotion engine analyzes the user's tone of voice, speech rate, intonation, etc. to identify the user's emotional state. Emotion engines used include IBM Watson Tone Analyzer and Lexalytics' Sentiment Analysis API. For example, if a user says "Is this really okay?" in an anxious voice, the emotion engine will detect the "anxiety."

[0624] 4. Fraud Pattern Detection

[0625] The server receives the text sent from the device and uses a generative artificial intelligence model to detect fraud-related patterns. This model compares past fraud datasets. Specifically, OpenAI's GPT-3 and Google BERT are used as generative technologies. If a user engages in a conversation that includes a phrase such as "Please tell me my account information," this is likely to be judged as fraud.

[0626] 5. Integrated judgment of emotional state and deception pattern

[0627] The server combines the emotion analysis results from the emotion engine with the fraud pattern detection results from the generative AI to make a comprehensive judgment. For example, if the emotion analysis result is "anxiety" and the fraud pattern matches, it will assess the possibility of fraud. This information is combined according to a pre-set algorithm by the system.

[0628] 6. Building and sending alerts

[0629] If it is determined that there is a high possibility of fraud, the server will construct an alert message. This message will include the detected fraud pattern, the user's emotional state, the detection date and time, and user information. The alert message will be sent to pre-registered family members and police agencies. In a specific implementation, the alert email will be sent using the SMTP protocol.

[0630] (Example)

[0631] The user is an elderly person who regularly communicates with family and the bank over the phone. One day, he receives a call saying, "Hello, is this the bank? I'd like to check my account information." During the conversation, the user says in an anxious voice, "Is it really safe?" The device that receives this conversation collects the voice data and converts it into text. At the same time, the emotion engine analyzes the tone and speed of the user's voice to detect feelings of anxiety.

[0632] The server receives the user's conversation content and emotional state and uses generative artificial intelligence to detect fraud-related patterns. For example, the phrase "I want to check my account information" is detected, and the emotion engine indicates that the user is in an anxious state. As a result, it is determined that there is a high possibility of fraud, and the server sends an alert message to the user's family and the police stating, "A conversation that may be fraudulent has been detected, and the user is in an anxious state."

[0633] The above is a concrete example of how to implement a fraud detection system that combines an emotion engine. This system will protect users from fraud risks with a higher degree of accuracy and enable quicker response by family members and the police.

[0634] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0635] Step 1:

[0636] The device uses a microphone to collect audio data from the user's surroundings. It then uses the device's built-in noise-canceling technology to capture clear audio data, reducing noise and providing highly accurate audio data. The collected audio data is temporarily stored in the device's memory.

[0637] Input: Ambient audio

[0638] Output: Clear audio data

[0639] Step 2:

[0640] The device sends the collected voice data to the Google Speech-to-Text API and converts it into text. During this process, the voice data is analyzed and corresponding text data is generated. The generated text data is then passed on to the next processing step.

[0641] Input: Clear voice data

[0642] Output: Character information (text data)

[0643] Step 3:

[0644] The device sends the text information to the emotion engine, which uses IBM Watson Tone Analyzer. This engine analyzes the tone, speed, and intonation of the words in the text data to identify the user's emotional state. The analysis results in an emotion label, such as "anxiety" or "joy."

[0645] Input: Character information (text data)

[0646] Output: Sentiment analysis results (emotion labels)

[0647] Step 4:

[0648] The device sends the text information and the results of the emotion analysis to the server. The server inputs the text information into a generative AI model as a prompt sentence. The prompt sentence is a specific sentence based on the text information. The generative AI model (e.g., GPT-3) analyzes the prompt sentence and detects fraud-related patterns. The detection result is a label such as "Fraud pattern detected" or "No problem."

[0649] Input: Text information, emotion analysis results

[0650] Output: Fraud-related pattern detection results

[0651] Step 5:

[0652] The server integrates the results of the sentiment analysis and the fraud-related pattern detection to make a comprehensive judgment. If the sentiment analysis result is "anxiety" and a fraud pattern is detected, the possibility of fraud is assessed as high. The result of this comprehensive judgment is used to construct a warning message.

[0653] Input: Sentiment analysis results, fraud-related pattern detection results

[0654] Output: Overall judgment result

[0655] Step 6:

[0656] The server then creates a warning message based on the comprehensive judgment results, which includes the detected fraud pattern, the user's emotional state, the detection date and time, user information, etc. This warning message is then sent to pre-registered family members and police agencies using the SMTP protocol.

[0657] Input: Overall judgment result

[0658] Output: Warning message

[0659] Step 7:

[0660] The server records detailed logs of all processing steps, including processing timestamps, sentiment analysis results, fraud pattern detection results, and details of any alert messages sent. This log is stored for later analysis and troubleshooting.

[0661] Input: All processing results

[0662] Output: Log data

[0663] (Application example 2)

[0664] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0665] The risk of elderly and socially vulnerable people falling victim to fraud continues to increase. Conventional fraud prevention systems have limitations in their ability to detect fraudulent patterns and do not adequately analyze the user's emotional state, making them unable to completely prevent fraud. There is a need for a system that can resolve these issues, more accurately and quickly identify potential fraud, and send alerts.

[0666] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0667] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text information, means for analyzing the user's emotional state based on the converted text information, generative artificial intelligence means for detecting fraud-related patterns using the analysis results, and means for sending an alert when a fraud-related pattern is detected. This makes it possible to comprehensively analyze the user's emotional state and fraud-related patterns, and to send an alert quickly and accurately when there is a high possibility of fraud.

[0668] The "means for collecting voice data" refers to a device that has the function of capturing and recording external voice in real time using a microphone on the terminal.

[0669] The "means for converting collected voice data into text information" is a device that has the function of analyzing voice data using voice recognition software and generating corresponding text data.

[0670] The "means for analyzing the user's emotional state based on the converted text information" is a device that has the function of analyzing the user's speaking tone, speed, choice of words, etc., and identifying the user's emotional state using an emotion engine.

[0671] A "generative artificial intelligence means" is an artificial intelligence system that has the ability to learn specific patterns or trends based on past data sets and detect those patterns or trends based on newly input data.

[0672] A "means for detecting fraud-related patterns" is a device capable of using generative artificial intelligence to identify phrases and behavioral patterns characteristic of fraudulent activity.

[0673] The "means for sending an alert" is a device that has the function of automatically sending an alert message to pre-registered recipients when there is a high possibility of fraud.

[0674] The present invention provides a system that collects voice data from a user, analyzes the data to detect the user's emotional state and fraud-related patterns, and sends alerts as needed.

[0675] 1. Collection of audio data

[0676] The device constantly collects voice data using a microphone, allowing it to capture conversations in real time as users interact. The collected voice data is also equipped with a noise filtering function to ensure clear voice data.

[0677] 2. Transcription of audio data

[0678] The collected voice data is converted into text using voice recognition software. For example, Google Speech-to-Text API is commonly used. This software analyzes the voice data and generates corresponding text data. For example, if a user says, "Is this from my bank?", that phrase is accurately recorded as text.

[0679] 3. Analysis by Emotion Engine

[0680] The server then uses an emotion engine to further analyze the converted voice data. The IBM Watson Tone Analyzer is often used as the emotion engine. This engine analyzes the tone, speed, and phrasing of the user's voice to identify the user's emotional state. For example, if the user says "Is it really safe?" in an anxious voice, the emotion engine will detect anxiety.

[0681] 4. Fraud Pattern Detection

[0682] Generative AI is used to detect fraud patterns from the converted text information. Examples of generative AI include OpenAI GPT-3. This method compares with past fraud datasets to identify fraud-related behavioral patterns and phrases. For example, if a conversation includes the phrase "Please tell me your account information," the system will determine that there is a high possibility of fraud.

[0683] 5. Sending an Alert

[0684] If it determines that there is a high possibility of fraud, the server will send a warning message to pre-registered contacts (family or police agencies). This message will include the detected fraud pattern, the user's emotional state, the date and time of detection, and user information. For example, a warning message will be sent stating, "A potentially fraudulent conversation has been detected and the user is in a state of anxiety."

[0685] 6. Logging

[0686] The server records a detailed log of all operations performed, including timestamps of operations, emotion engine analysis results, fraud pattern detection results, details of alert messages sent, etc. This log can be used for later analysis and troubleshooting.

[0687] Specific examples

[0688] For example, suppose an elderly person receives a phone call saying, "Hello, is this the bank? I'd like to check my account information." The device receiving this conversation collects voice data and converts it into text. At the same time, an emotion engine analyzes the tone and speed of the user's voice to detect feelings of anxiety. The content of the user's conversation and emotional state are sent to a server, which uses generative artificial intelligence to detect fraud-related patterns. For example, if the phrase "I'd like to check my account information" is detected and the emotion engine indicates that the user is in an anxious state, it is determined that there is a high possibility of fraud. As a result, the server sends an alert message to the family and police stating, "A potentially fraudulent conversation has been detected and the user is in an anxious state."

[0689] Prompt Sentence Examples

[0690] Determine if the following text contains any signs of fraud: "Hello, is this your bank? I'd like to verify my account information."

[0691] This invention provides users with high accuracy protection from fraud and can send alerts quickly, allowing for a quick response by family members or police.

[0692] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0693] Step 1:

[0694] The device uses a microphone to collect surrounding audio data, which is then saved in real time and filtered to ensure clear audio.

[0695] Input: External audio

[0696] Output: filtered audio data

[0697] Step 2:

[0698] The filtered voice data is converted into text using voice recognition software (e.g., Google Speech-to-Text API).

[0699] Input: filtered audio data

[0700] Output: Character information (text data)

[0701] Step 3:

[0702] The text information is sent to a server, and the user's emotional state is analyzed using an emotion engine (e.g., IBM Watson Tone Analyzer). The emotion engine analyzes the tone, speed, and phrasing of the text data to identify the user's emotion.

[0703] Input: Character information (text data)

[0704] Output: Emotion analysis results (anxiety, anger, joy, etc.)

[0705] Step 4:

[0706] The server uses generative AI (e.g., OpenAI GPT-3) to detect fraud-related patterns based on the sentiment analysis results and text information. This AI compares the results with fraud patterns learned from past datasets to find signs of fraud.

[0707] Input: Emotion analysis results, character information (text data)

[0708] Output: Fraud pattern detection results

[0709] Step 5:

[0710] If a fraudulent activity is detected, the server sends a warning message to pre-registered contacts, which includes information about the detected fraudulent activity, the user's emotional state, the date and time of the detection, and other user information.

[0711] Input: Fraud pattern detection results, sentiment analysis results

[0712] Output: Warning message

[0713] Step 6:

[0714] The server records detailed logs of all operations performed, including timestamps of operations, emotion engine analysis results, fraud pattern detection results, and details of any alert messages sent, for later analysis and troubleshooting.

[0715] Input: Output results of each step

[0716] Output: Detailed log information

[0717] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0718] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0719] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0720] [Third embodiment]

[0721] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0722] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0723] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0724] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0725] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0726] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0727] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0728] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0729] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0730] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0731] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0732] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0733] 1. Collection of audio data

[0734] The device constantly monitors surrounding voice data using a microphone. The device also has a voice recognition sensor that collects user conversations in real time. For example, voice data is acquired when the user is talking on the phone or talking face-to-face with family members.

[0735] 2. Transcription of audio data

[0736] The device uses software to convert the collected voice data into text. This software uses speech recognition technology to analyze the voice data and generate corresponding text. For example, if a user says, "Is this from my bank?", this phrase will be recorded as text.

[0737] 3. Fraud Pattern Detection

[0738] The server receives text sent from the device and uses generative artificial intelligence to detect fraud-related patterns. This AI model is trained on historical fraud datasets and determines in real time whether the user's conversation matches the characteristics of fraud. For example, if a phrase like "Can you tell me my account information?" is detected in the conversation, it will mark it as a possible fraud.

[0739] 4. Sending alerts

[0740] If a fraud-related pattern is detected, the server immediately sends an alert. Recipients of the alert include pre-registered family members and police agencies. For example, if a user utters a typical fraudulent phrase like "Do I need to provide my account number?", the server will send an alert message to family members and the police saying, "A possible fraudulent conversation has been detected."

[0741] Specific examples

[0742] User Status

[0743] The user is an elderly person who regularly communicates with his family and bank over the phone. One day, he receives a call saying, "Hello, is this the bank? I'd like to check my account information."

[0744] Device behavior

[0745] The terminal that receives this conversation immediately collects the voice data and converts it into text information.

[0746] Server Operation

[0747] The user's conversation is sent to a server, which uses generative artificial intelligence to detect fraud patterns. In this case, the phrase "I would like to check my account information" matches the fraud pattern, so it is immediately determined to be a potential fraud.

[0748] Sending alerts

[0749] If a fraudulent conversation is detected, the server sends a warning message to the user's family and the police, saying, "A potential fraudulent conversation has been detected." This allows the family and the police to respond quickly and prevent the user from becoming a victim of fraud.

[0750] The above is a specific embodiment for carrying out the present invention. The present invention protects users from the risk of fraud and allows them to quickly receive support from their families or the police.

[0751] The processing flow will be explained below.

[0752] Specific steps for collecting and processing audio data

[0753] Step 1: Collecting audio data

[0754] The device activates voice recognition software and captures surrounding audio through a microphone, monitoring all user conversations in real time while filtering out background noise and other distracting sounds.

[0755] Step 2: Temporarily save the audio data

[0756] The device temporarily stores the collected voice data in its internal memory. This data is temporarily held for subsequent processing. As voice data is collected, it is accumulated in a buffer.

[0757] Step 3: Transcribe the audio data

[0758] The device uses voice recognition software to convert the temporarily stored voice data into text. The text generated by this process is a textual representation of what the user said. For example, "Hello, is this from my bank?"

[0759] Step 4: Send text information

[0760] The device then sends the converted text data, including necessary metadata (such as a timestamp and user ID), to a server via the Internet.

[0761] Step 5: Detect fraud patterns

[0762] The server analyzes the received text and uses generative artificial intelligence to detect fraud patterns. Specifically, it compares it with historical data sets and identifies phrases and wording characteristic of fraud. This analysis is done in real time.

[0763] Step 6: Determine the likelihood of fraud

[0764] The server evaluates the likelihood of fraud based on the results of the generative AI. If the algorithm determines that there is a high risk of fraud, the result is recorded.

[0765] Step 7: Building the Alert

[0766] If the server detects possible fraud, it constructs an alert message that includes the nature of the detected fraud, the date and time, and the affected user.

[0767] Step 8: Sending an alert

[0768] The server sends alert messages to pre-registered family members and police agencies via email and SMS, allowing recipients to receive the alerts immediately and respond quickly.

[0769] Step 9: Logging

[0770] The server records detailed logs of all operations it performs, including timestamps of operations, fraud patterns detected, and details of warning messages sent, for later analysis and troubleshooting.

[0771] Example 1

[0772] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0773] Today, the risk of becoming a victim of fraud is increasing, especially among the elderly, and there is a need to detect signs of fraud in real time and respond quickly. Furthermore, as fraud methods become more sophisticated, it is difficult to respond with traditional manual monitoring and vigilance, and cooperation from family members and police agencies is essential. Therefore, it is necessary to realize a fraud detection system that uses voice data.

[0774] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0775] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text information, means for transmitting the converted text information to the server, generative artificial intelligence means for detecting fraud-related patterns using the text information received by the server, and means for sending an alert when a fraud-related pattern is detected. This makes it possible to detect signs of fraud in real time, send an alert quickly, and protect users from fraud.

[0776] "Voice data" refers to information that is a digital recording of the user's speech and surrounding sounds.

[0777] "Collection means" refers to a device or method for acquiring audio data using a microphone, sensor, etc.

[0778] "Text information" refers to text data converted from collected voice data using voice recognition technology.

[0779] "Means of converting into textual information" refers to the process of converting audio data into textual data using speech recognition software or algorithms.

[0780] The "means for transmitting the converted character information" is a mechanism for transferring the generated text data to a server via a secure protocol.

[0781] "Generative artificial intelligence" is an AI technology that learns from past data sets and makes predictions and judgments in real time about new data.

[0782] "Fraud-related patterns" refer to specific phrases or patterns of behavior that indicate fraudulent activity.

[0783] "Means for sending an alert" refers to a method for sending a warning message to pre-registered recipients when a fraud-related pattern is detected.

[0784] A "historical fraud dataset" is a collection of previously recorded fraud cases and patterns.

[0785] "Family and Law Enforcement" refers to contacts that include the user's pre-registered relatives and law enforcement agencies.

[0786] MODE FOR CARRYING OUT THE INVENTION

[0787] The present invention relates to a system that uses specific hardware and software to provide real-time fraud protection to users, and a specific embodiment of the system is described below.

[0788] Audio data collection

[0789] The device constantly collects surrounding audio data using a microphone. The device is equipped with a high-performance voice recognition sensor that captures the user's conversation in real time. The device can be a smartphone, tablet, smart speaker, or other device. For example, when a user is talking on the phone, the device can be equipped with a function to record the conversation.

[0790] Transcription of audio data

[0791] The device uses speech recognition software to convert collected voice data into text. Specifically, it uses speech recognition technologies such as the Google Speech-to-Text API and Microsoft Azure Cognitive Services Speech API. This software analyzes the voice data and generates corresponding text. For example, if a user says, "Is this from my bank?", this phrase will be recorded as text on the device.

[0792] Sending audio data

[0793] The device sends the generated text data to the server in an encrypted format using a secure protocol (e.g., HTTPS), which ensures data security and privacy.

[0794] Fraud pattern detection

[0795] The server receives the text sent from the device and begins analyzing it in real time using generative artificial intelligence (e.g., OpenAI GPT). The software on the server is trained on historical fraud datasets and determines whether the text matches the characteristics of fraud. For example, if the phrase "Please tell me your account information" is detected, it will be marked as a possible fraud.

[0796] Sending alerts

[0797] If a fraud-related pattern is detected, the server immediately sends an alert. Recipients of the alert include pre-registered family members and police agencies. For example, if a user says, "Do I need to provide my account number?", the server will send an alert message to family members and the police saying, "A potentially fraudulent conversation has been detected."

[0798] Specific examples

[0799] User Status

[0800] The user is an elderly person who regularly communicates with family and financial institutions over the phone. One day, he receives a call saying, "Hello, is this the bank? I'd like to check my account information."

[0801] Device behavior

[0802] The device that receives this conversation immediately collects the voice data and converts it into text information. The device is a smartphone, and the voice data is converted into text using the Google Speech-to-Text API.

[0803] Server Operation

[0804] The user's conversation is sent to a server, which uses generative artificial intelligence to detect fraud patterns. In this case, the phrase "I would like to check my account information" matches the fraud pattern, so it is immediately determined to be a potential fraud.

[0805] Sending alerts

[0806] If a fraudulent conversation is detected, the server sends a warning message to the user's family and the police, saying, "A potential fraudulent conversation has been detected." This allows the family and the police to respond quickly and prevent the user from becoming a victim of fraud.

[0807] Prompt Sentence Examples

[0808] "If a user calls and says, 'Is this your bank? I'd like to confirm my account information,' we'll mark it as a fraud pattern and send an alert message to their family and the police."

[0809] This explains the configuration of a system for specifically implementing the present invention, which protects users from the risk of fraud and sends prompt alerts to obtain cooperation from family members and the police.

[0810] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0811] Step 1:

[0812] The device constantly collects surrounding audio data using a microphone. When the user speaks, the device captures the audio and records it in digital form. The input is the user's speech, and the output is digital audio data. For example, a user might record themselves saying, "Hello, is this the bank?"

[0813] Step 2:

[0814] The device converts the collected voice data into text using the Google Speech-to-Text API. During this conversion process, voice recognition technology is used to analyze the voice data and generate corresponding text. The input is digital voice data, and the output is text data. Specifically, the voice saying "Hello, is this the bank?" is converted into text "Hello, is this the bank?"

[0815] Step 3:

[0816] The terminal securely transmits the generated text data to the server using the HTTPS protocol. During the data transmission process, data security and privacy are guaranteed by encryption. The input is text data, and the output is secure data transmission to the server. Specifically, the text data "Hello, is this the bank?" is transmitted to the server.

[0817] Step 4:

[0818] The server analyzes the received text data in real time. It runs a generative artificial intelligence (e.g., OpenAI GPT) and compares it with past fraud datasets to detect fraud-related patterns. The input is text data, and the output is the detection result of fraud-related patterns. For example, if the phrase "Please tell me your account information" is detected, it is marked as a possible fraud.

[0819] Step 5:

[0820] If a fraud-related pattern is detected, the server immediately generates an alert and sends it to pre-registered family members and police agencies. During this process, the generated alert message is forwarded to each contact in the distribution list. The input is a fraud-related text phrase, and the output is the alert message sent to family members and police agencies. Specifically, the alert message sent is "A potential fraudulent conversation has been detected."

[0821] Through these steps, the system protects users from the risk of fraud and can quickly issue alerts to prompt action from relevant authorities.

[0822] (Application example 1)

[0823] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0824] In modern society, cases of elderly people falling victim to telephone fraud are increasing. To prevent such fraud, real-time fraud detection and rapid response methods are required. However, conventional methods are insufficient to efficiently monitor telephone conversations and detect possible fraud, making it difficult to completely prevent fraud. Technology to address this issue is needed.

[0825] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0826] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text information, generative artificial intelligence means for detecting fraud-related patterns using the converted text information, means for sending an alert when a fraud-related pattern is detected, means for converting voice data into text using voice recognition software, means for analyzing fraud patterns using a generative artificial intelligence model using a cloud service, and notification means for sending an emergency notification when fraud is detected. This makes it possible to detect potentially fraudulent telephone conversations in real time and quickly notify family members and the police.

[0827] "Audio data" refers to information that records audio in digital format.

[0828] "Means of collection" refers to devices and software for acquiring and storing voice data.

[0829] "Means for converting into textual information" refers to technology or software that converts audio data into text format.

[0830] "Generative artificial intelligence" refers to an AI model that learns from past data and analyzes new data.

[0831] "Detecting fraud-related patterns" is the process of determining whether certain phrases or behaviors are associated with fraud.

[0832] An "alert means" is a system or technology for providing notification in the event of suspected fraud.

[0833] "Speech recognition software" is a program that converts speech into text in real time.

[0834] "Cloud services" are computer resources and data analysis services provided via the Internet.

[0835] A "generative artificial intelligence model" is an AI model used to detect fraud patterns from text data.

[0836] A "notification mechanism" is a system capable of sending emergency notifications based on detected fraud patterns.

[0837] This invention is a real-time fraud detection and emergency notification system that monitors users' phone conversations using their smartphones and can take immediate action if fraudulent activity is suspected.

[0838] System configuration

[0839] The system consists of the following main components:

[0840] 1. User device (smartphone)

[0841] 2. Speech recognition software (Google Speech-to-Text API)

[0842] 3. Cloud Services (Amazon Web Services: AWS Lambda and AWS SNS)

[0843] 4. Generative AI models (e.g., GPT-4)

[0844] Hardware and Software Use

[0845] The user device is equipped with a built-in microphone that is used to collect real-time phone conversations, and this voice data is converted into text using the Google Speech-to-Text API.

[0846] The converted text is sent to AWS Lambda, a cloud service, and analyzed by a generative AI model (e.g., GPT-4), which is trained on historical fraud datasets to detect phrases and patterns characteristic of fraud.

[0847] If a suspicious pattern is detected, AWS SNS will send an emergency notification to pre-registered family members and law enforcement agencies, including a message such as "A potential fraudulent conversation has been detected."

[0848] Specific examples of processing

[0849] User Status

[0850] The user is an elderly person who regularly communicates with family, banks, etc. by telephone. One day, he receives a call saying, "Hello, this is the bank. I would like to confirm my account information."

[0851] Device behavior

[0852] The user's device collects this conversation in real time and converts the audio data into text using the Google Speech-to-Text API.

[0853] Server Operation

[0854] The converted text is sent to AWS Lambda in the cloud and analyzed using a generative artificial intelligence model (e.g., GPT-4). If the detected patterns match the characteristics of a fraud, an emergency notification is sent to family members and police via AWS SNS.

[0855] Prompt Sentence Examples

[0856] Determine whether the following conversation is likely to be a scam and, if so, raise a red flag.

[0857] Conversation: This is a bank. Please let me know your account number so I can process your request.

[0858] Risk of fraud:

[0859] The system protects users from the risk of fraud and allows them to quickly access family or police assistance.

[0860] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0861] Step 1:

[0862] A user terminal collects the telephone conversation.

[0863] Specific operation: When a user starts a call, voice data is collected in real time using the smartphone's built-in microphone.

[0864] Input: Audio data from a user's phone conversation.

[0865] Output: The audio data is saved to the device.

[0866] Step 2:

[0867] The device converts the collected voice data into text information.

[0868] What it does: The audio data is converted to text using the Google Speech-to-Text API.

[0869] Input: The audio data collected in step 1.

[0870] Output: Textual information in text format.

[0871] Step 3:

[0872] The terminal transmits character information in text format to the server.

[0873] Specific operation: The converted text information is uploaded to a server via the Internet.

[0874] Input: The textual information generated in step 2.

[0875] Output: Textual information stored on the server.

[0876] Step 4:

[0877] The server analyzes the text information using a generative artificial intelligence model.

[0878] Specific operation: A generative artificial intelligence model (such as GPT-4) is launched on the server side using AWS Lambda and analyzes text information.

[0879] Input: The textual information sent to the server in step 3.

[0880] Output: Fraud probability assessment results.

[0881] Step 5:

[0882] The server detects fraud patterns.

[0883] How it works: Generative AI models detect phrases and patterns characteristic of fraud.

[0884] Input: The text parsed in step 4.

[0885] Output: Judgment result on whether or not there is suspicion of fraud.

[0886] Step 6:

[0887] If a fraud pattern is detected, the server sends an emergency notification.

[0888] What it does: Uses AWS SNS to send emergency notifications to pre-registered family members and police agencies.

[0889] Input: The probability of fraud determined in step 5.

[0890] Output: Urgent notification message (e.g. "Potential fraudulent conversation detected").

[0891] Step 7:

[0892] Family members and police are notified and begin responding.

[0893] Specific actions: Family members or police who receive the notification will contact the user and take action.

[0894] Input: The emergency notification sent in step 6.

[0895] Output: Ensuring user safety and implementing fraud prevention measures.

[0896] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0897] 1. Collection of audio data

[0898] The device constantly collects surrounding audio data using a microphone. When a user is having a phone call or a face-to-face conversation, the audio data is monitored in real time. This data collection is equipped with a noise filtering function to ensure clear audio data.

[0899] 2. Transcription of audio data

[0900] The device runs speech recognition software to convert the collected voice data into text. The software analyzes the speech and generates corresponding text data. For example, if a user says, "Is this from my bank?", that phrase is recorded verbatim as text.

[0901] 3. Analysis by Emotion Engine

[0902] The device uses an emotion engine to further analyze the converted voice data. The emotion engine analyzes the tone, speed, and phrasing of the user's voice to identify the user's emotional state. For example, if the user says "Are you sure it's safe?" in an anxious voice, the emotion engine will detect anxiety and fear.

[0903] 4. Fraud Pattern Detection

[0904] The server receives text sent from the device and uses generative artificial intelligence to detect fraud-related patterns. The model compares it with historical fraud datasets to identify phrases and behavioral patterns characteristic of fraud. If a user engages in a conversation that includes the phrase "Please tell me my account information," it is determined to be a potential fraud.

[0905] 5. Integrated judgment of emotional state and deception pattern

[0906] The server combines the analysis results from the emotion engine and the fraud pattern detection results from the generative AI to make a comprehensive judgment. If the user's emotional state is biased toward anxiety or fear, and if fraud-related patterns are detected, the server will assess the possibility of fraud as high.

[0907] 6. Building and sending alerts

[0908] If a fraudulent conversation is deemed likely, the server creates a warning message. This message includes the detected fraudulent pattern, the user's emotional state, the date and time of the detection, and the user's information. The warning message is sent to pre-registered family members and police agencies. For example, a message stating "A potentially fraudulent conversation has been detected and the user is in a state of anxiety" is sent.

[0909] 7. Logging

[0910] The server records detailed logs of all operations performed, including timestamps of operations, emotion engine analysis results, fraud pattern detection results, and details of any alert messages sent, for later analysis and troubleshooting.

[0911] Specific examples

[0912] User Status

[0913] The user is an elderly person who regularly communicates with his family and bank over the phone. One day, he received a call saying, "Hello, is this your bank? I'd like to confirm my account information." During the conversation, the user asked in an anxious voice, "Is this really safe?"

[0914] Device behavior

[0915] The device that receives this conversation collects the voice data and converts it into text information. At the same time, the emotion engine analyzes the tone and speed of the user's voice to detect feelings of anxiety.

[0916] Server Operation

[0917] The user's conversational content and emotional state are sent to the server, which uses generative artificial intelligence to detect fraud-related patterns. For example, if the phrase "I want to check my account information" is detected, and the emotion engine indicates that the user is in an anxious state, it will determine that there is a high possibility of fraud.

[0918] Sending an alert

[0919] If it is determined that there is a high possibility of fraud, the server will send a warning message to the family and the police stating that "a potentially fraudulent conversation has been detected and the user is in a state of anxiety."

[0920] The above is a concrete example of how to implement the present invention by combining an emotion engine. This system protects users from the risk of fraud with a higher degree of accuracy and enables quicker response by family members and the police.

[0921] The processing flow will be explained below.

[0922] Specific steps for collecting and analyzing audio data

[0923] Step 1: Collecting audio data

[0924] The device uses a built-in microphone to capture the user's surrounding sounds in real time, recording all conversations whether the user is making a phone call or talking face-to-face. The device also has a filtering function to reduce environmental noise, ensuring clear audio data is collected.

[0925] Step 2: Temporarily save the audio data

[0926] The device temporarily stores the collected voice data in its internal memory. This ensures stable storage of the voice data and makes it available for subsequent processing. Because the voice data is temporarily stored as a buffer, processing can proceed without interruption.

[0927] Step 3: Transcribe the audio data

[0928] The device uses voice recognition software to analyze the temporarily stored voice data and convert it into text data. For example, a user might say, "Hello, is this from my bank?" and the transcription is then adjusted for accuracy.

[0929] Step 4: Send text information

[0930] The device sends the converted text data and associated metadata (such as user ID and timestamp) to the server, which then promptly transmits the data over the internet connection.

[0931] Step 5: Analyze emotional state

[0932] The server analyzes the voice data received from the device using an emotion engine to evaluate the user's emotional state. The emotion engine analyzes the tone, speed, and strength of the voice to detect emotions such as "anxiety" and "fear." For example, if a user says "Is it really safe?" in a tense voice, the emotion of anxiety is detected.

[0933] Step 6: Detect fraud patterns

[0934] The server then inputs the received text data, along with the emotional state, into a generative AI model to detect fraud-related patterns. The AI ​​model then matches it with a large amount of historical data to identify phrases and phrasing associated with fraudulent activity. For example, the phrase "Please tell me your account information" might be detected.

[0935] Step 7: Comprehensive Decision

[0936] The server integrates the analysis results of the emotion engine and the fraud pattern detection results to comprehensively evaluate the risk of fraud. If the user's emotional state is evaluated as "anxiety" or "fear" and also matches the characteristic fraud patterns, it is determined that there is a very high possibility of fraud.

[0937] Step 8: Building the Alert

[0938] If a fraudulent conversation is detected, the server immediately creates an alert message, which includes the details of the detected fraud, the user's emotional state, the date and time, the user ID, etc. For example, the alert might say, "A potentially fraudulent conversation has been detected and the user is in a state of anxiety."

[0939] Step 9: Sending an alert

[0940] The server then sends the constructed warning message to pre-registered family members and police agencies via email and SMS, ensuring that recipients receive the warning quickly.

[0941] Step 10: Logging

[0942] The server records detailed logs of all operations performed, including timestamps of operations, emotion engine and generative AI results, and details of warning messages. This log data can be used for future analysis and troubleshooting.

[0943] Specific examples

[0944] User Status

[0945] The user is an elderly person who regularly has phone and face-to-face conversations. One day, he received a call saying, "Hello, is this your bank? I'd like to confirm my account information." During the conversation, the user said in an anxious voice, "Is this really safe?"

[0946] Device behavior

[0947] The device that captures this conversation collects the voice data, transcribes it, and immediately sends the results to the server. At the same time, the emotion engine detects anxiety from the user's tone of voice.

[0948] Server Operation

[0949] Based on the received text data and the user's emotional state, the server uses generative artificial intelligence to detect fraud patterns. For example, if the phrase "I want to check my account information" matches the characteristics of a fraud, and the user's anxiety is detected, the server determines that there is a high possibility of fraud.

[0950] Sending an alert

[0951] Because it is determined that there is a high possibility of fraud, the server sends a warning message to the family and the police stating that "a potentially fraudulent conversation has been detected and the user is in a state of anxiety."

[0952] This provides users with far greater protection from the risk of fraud and allows for faster response.

[0953] Example 2

[0954] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0955] In modern society, voice-mediated fraud is on the rise, and elderly people and those who are not familiar with information are particularly vulnerable to becoming victims. Conventional fraud detection systems have technologies to recognize fraudulent patterns, but they have limitations in their methods of analyzing the user's emotional state in real time and making an integrated judgment to further increase the likelihood of fraud. This has led to frequent false positives, increasing the possibility that appropriate warnings will not be issued in situations where protection is truly needed.

[0956] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text information, generative artificial intelligence means for detecting fraud-related patterns using the converted text information, emotion analysis means for analyzing the voice data to identify the user's emotional state, and means for integrating the analysis results of the emotion analysis means and the fraud-related pattern detection results to make a comprehensive judgment. This enables highly accurate fraud detection that takes the user's emotional state into consideration, and effectively protects users from the risk of fraud.

[0957] "Audio data" is signal information including the user's and surrounding sounds.

[0958] "Text information" is information in text format obtained by analyzing voice data.

[0959] "Generative AI methods" are AI techniques that learn from past data and generate and identify specific patterns based on newly input data.

[0960] "Emotion analysis means" refers to technology and devices that identify the user's emotional state from voice data and text information.

[0961] "Fraud-related patterns" refer to specific phrases or behavioral characteristics that characterize fraudulent activity.

[0962] "Means for making a comprehensive judgment" refers to the technology and process that integrates the results of sentiment analysis and fraud-related pattern detection to make a final judgment.

[0963] "Means for sending an alert" means the functions and techniques for constructing and sending an alert message to a designated recipient when fraud is determined to be likely.

[0964] The "historical fraud dataset" is a database that collects data on fraud cases that have occurred in the past.

[0965] "Pre-registered family members and police agencies" are emergency contacts that the user has preset in the system.

[0966] This system collects and analyzes user voice data to detect and warn of fraudulent activity early. The following hardware and software are used to implement this system:

[0967] 1. Collection of audio data

[0968] The device uses a microphone to constantly collect sounds around the user. Noise-canceling technology is used to ensure clear audio data. Specifically, a smartphone or a dedicated audio collection device is used.

[0969] 2. Transcription of audio data

[0970] The device runs speech recognition software, such as the Google Speech-to-Text API or Microsoft Azure Cognitive Services, to convert the collected voice data into text. For example, if a user says, "Is this from my bank?", the exact phrase is recorded as text data.

[0971] 3. Analysis by Emotion Engine

[0972] The device runs an emotion engine to analyze the voice data converted into text. The emotion engine analyzes the user's tone of voice, speech rate, intonation, etc. to identify the user's emotional state. Emotion engines used include IBM Watson Tone Analyzer and Lexalytics' Sentiment Analysis API. For example, if a user says "Is this really okay?" in an anxious voice, the emotion engine will detect the "anxiety."

[0973] 4. Fraud Pattern Detection

[0974] The server receives the text sent from the device and uses a generative artificial intelligence model to detect fraud-related patterns. This model compares past fraud datasets. Specifically, OpenAI's GPT-3 and Google BERT are used as generative technologies. If a user engages in a conversation that includes a phrase such as "Please tell me my account information," this is likely to be judged as fraud.

[0975] 5. Integrated judgment of emotional state and deception pattern

[0976] The server combines the emotion analysis results from the emotion engine with the fraud pattern detection results from the generative AI to make a comprehensive judgment. For example, if the emotion analysis result is "anxiety" and the fraud pattern matches, it will assess the possibility of fraud. This information is combined according to a pre-set algorithm by the system.

[0977] 6. Building and sending alerts

[0978] If it is determined that there is a high possibility of fraud, the server will construct an alert message. This message will include the detected fraud pattern, the user's emotional state, the detection date and time, and user information. The alert message will be sent to pre-registered family members and police agencies. In a specific implementation, the alert email will be sent using the SMTP protocol.

[0979] (Example)

[0980] The user is an elderly person who regularly communicates with family and the bank over the phone. One day, he receives a call saying, "Hello, is this the bank? I'd like to check my account information." During the conversation, the user says in an anxious voice, "Is it really safe?" The device that receives this conversation collects the voice data and converts it into text. At the same time, the emotion engine analyzes the tone and speed of the user's voice to detect feelings of anxiety.

[0981] The server receives the user's conversation content and emotional state and uses generative artificial intelligence to detect fraud-related patterns. For example, the phrase "I want to check my account information" is detected, and the emotion engine indicates that the user is in an anxious state. As a result, it is determined that there is a high possibility of fraud, and the server sends an alert message to the user's family and the police stating, "A conversation that may be fraudulent has been detected, and the user is in an anxious state."

[0982] The above is a concrete example of how to implement a fraud detection system that combines an emotion engine. This system will protect users from fraud risks with a higher degree of accuracy and enable quicker response by family members and the police.

[0983] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0984] Step 1:

[0985] The device uses a microphone to collect audio data from the user's surroundings. It then uses the device's built-in noise-canceling technology to capture clear audio data, reducing noise and providing highly accurate audio data. The collected audio data is temporarily stored in the device's memory.

[0986] Input: Ambient audio

[0987] Output: Clear audio data

[0988] Step 2:

[0989] The device sends the collected voice data to the Google Speech-to-Text API and converts it into text. During this process, the voice data is analyzed and corresponding text data is generated. The generated text data is then passed on to the next processing step.

[0990] Input: Clear voice data

[0991] Output: Character information (text data)

[0992] Step 3:

[0993] The device sends the text information to the emotion engine, which uses IBM Watson Tone Analyzer. This engine analyzes the tone, speed, and intonation of the words in the text data to identify the user's emotional state. The analysis results in an emotion label, such as "anxiety" or "joy."

[0994] Input: Character information (text data)

[0995] Output: Sentiment analysis results (emotion labels)

[0996] Step 4:

[0997] The device sends the text information and the results of the emotion analysis to the server. The server inputs the text information into a generative AI model as a prompt sentence. The prompt sentence is a specific sentence based on the text information. The generative AI model (e.g., GPT-3) analyzes the prompt sentence and detects fraud-related patterns. The detection result is a label such as "Fraud pattern detected" or "No problem."

[0998] Input: Text information, emotion analysis results

[0999] Output: Fraud-related pattern detection results

[1000] Step 5:

[1001] The server integrates the results of the sentiment analysis and the fraud-related pattern detection to make a comprehensive judgment. If the sentiment analysis result is "anxiety" and a fraud pattern is detected, the possibility of fraud is assessed as high. The result of this comprehensive judgment is used to construct a warning message.

[1002] Input: Sentiment analysis results, fraud-related pattern detection results

[1003] Output: Overall judgment result

[1004] Step 6:

[1005] The server then creates a warning message based on the comprehensive judgment results, which includes the detected fraud pattern, the user's emotional state, the detection date and time, user information, etc. This warning message is then sent to pre-registered family members and police agencies using the SMTP protocol.

[1006] Input: Overall judgment result

[1007] Output: Warning message

[1008] Step 7:

[1009] The server records detailed logs of all processing steps, including processing timestamps, sentiment analysis results, fraud pattern detection results, and details of any alert messages sent. This log is stored for later analysis and troubleshooting.

[1010] Input: All processing results

[1011] Output: Log data

[1012] (Application example 2)

[1013] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1014] The risk of elderly and socially vulnerable people falling victim to fraud continues to increase. Conventional fraud prevention systems have limitations in their ability to detect fraudulent patterns and do not adequately analyze the user's emotional state, making them unable to completely prevent fraud. There is a need for a system that can resolve these issues, more accurately and quickly identify potential fraud, and send alerts.

[1015] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1016] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text information, means for analyzing the user's emotional state based on the converted text information, generative artificial intelligence means for detecting fraud-related patterns using the analysis results, and means for sending an alert when a fraud-related pattern is detected. This makes it possible to comprehensively analyze the user's emotional state and fraud-related patterns, and to send an alert quickly and accurately when there is a high possibility of fraud.

[1017] The "means for collecting voice data" refers to a device that has the function of capturing and recording external voice in real time using a microphone on the terminal.

[1018] The "means for converting collected voice data into text information" is a device that has the function of analyzing voice data using voice recognition software and generating corresponding text data.

[1019] The "means for analyzing the user's emotional state based on the converted text information" is a device that has the function of analyzing the user's speaking tone, speed, choice of words, etc., and identifying the user's emotional state using an emotion engine.

[1020] A "generative artificial intelligence means" is an artificial intelligence system that has the ability to learn specific patterns or trends based on past data sets and detect those patterns or trends based on newly input data.

[1021] A "means for detecting fraud-related patterns" is a device capable of using generative artificial intelligence to identify phrases and behavioral patterns characteristic of fraudulent activity.

[1022] The "means for sending an alert" is a device that has the function of automatically sending an alert message to pre-registered recipients when there is a high possibility of fraud.

[1023] The present invention provides a system that collects voice data from a user, analyzes the data to detect the user's emotional state and fraud-related patterns, and sends alerts as needed.

[1024] 1. Collection of audio data

[1025] The device constantly collects voice data using a microphone, allowing it to capture conversations in real time as users interact. The collected voice data is also equipped with a noise filtering function to ensure clear voice data.

[1026] 2. Transcription of audio data

[1027] The collected voice data is converted into text using voice recognition software. For example, Google Speech-to-Text API is commonly used. This software analyzes the voice data and generates corresponding text data. For example, if a user says, "Is this from my bank?", that phrase is accurately recorded as text.

[1028] 3. Analysis by Emotion Engine

[1029] The server then uses an emotion engine to further analyze the converted voice data. The IBM Watson Tone Analyzer is often used as the emotion engine. This engine analyzes the tone, speed, and phrasing of the user's voice to identify the user's emotional state. For example, if the user says "Is it really safe?" in an anxious voice, the emotion engine will detect anxiety.

[1030] 4. Fraud Pattern Detection

[1031] Generative AI is used to detect fraud patterns from the converted text information. Examples of generative AI include OpenAI GPT-3. This method compares with past fraud datasets to identify fraud-related behavioral patterns and phrases. For example, if a conversation includes the phrase "Please tell me your account information," the system will determine that there is a high possibility of fraud.

[1032] 5. Sending an Alert

[1033] If it determines that there is a high possibility of fraud, the server will send a warning message to pre-registered contacts (family or police agencies). This message will include the detected fraud pattern, the user's emotional state, the date and time of detection, and user information. For example, a warning message will be sent stating, "A potentially fraudulent conversation has been detected and the user is in a state of anxiety."

[1034] 6. Logging

[1035] The server records a detailed log of all operations performed, including timestamps of operations, emotion engine analysis results, fraud pattern detection results, details of alert messages sent, etc. This log can be used for later analysis and troubleshooting.

[1036] Specific examples

[1037] For example, suppose an elderly person receives a phone call saying, "Hello, is this the bank? I'd like to check my account information." The device receiving this conversation collects voice data and converts it into text. At the same time, an emotion engine analyzes the tone and speed of the user's voice to detect feelings of anxiety. The content of the user's conversation and emotional state are sent to a server, which uses generative artificial intelligence to detect fraud-related patterns. For example, if the phrase "I'd like to check my account information" is detected and the emotion engine indicates that the user is in an anxious state, it is determined that there is a high possibility of fraud. As a result, the server sends an alert message to the family and police stating, "A potentially fraudulent conversation has been detected and the user is in an anxious state."

[1038] Prompt Sentence Examples

[1039] Determine if the following text contains any signs of fraud: "Hello, is this your bank? I'd like to verify my account information."

[1040] This invention provides users with high accuracy protection from fraud and can send alerts quickly, allowing for a quick response by family members or police.

[1041] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1042] Step 1:

[1043] The device uses a microphone to collect surrounding audio data, which is then saved in real time and filtered to ensure clear audio.

[1044] Input: External audio

[1045] Output: filtered audio data

[1046] Step 2:

[1047] The filtered voice data is converted into text using voice recognition software (e.g., Google Speech-to-Text API).

[1048] Input: filtered audio data

[1049] Output: Character information (text data)

[1050] Step 3:

[1051] The text information is sent to a server, and the user's emotional state is analyzed using an emotion engine (e.g., IBM Watson Tone Analyzer). The emotion engine analyzes the tone, speed, and phrasing of the text data to identify the user's emotion.

[1052] Input: Character information (text data)

[1053] Output: Emotion analysis results (anxiety, anger, joy, etc.)

[1054] Step 4:

[1055] The server uses generative AI (e.g., OpenAI GPT-3) to detect fraud-related patterns based on the sentiment analysis results and text information. This AI compares the results with fraud patterns learned from past datasets to find signs of fraud.

[1056] Input: Emotion analysis results, character information (text data)

[1057] Output: Fraud pattern detection results

[1058] Step 5:

[1059] If a fraudulent activity is detected, the server sends a warning message to pre-registered contacts, which includes information about the detected fraudulent activity, the user's emotional state, the date and time of the detection, and other user information.

[1060] Input: Fraud pattern detection results, sentiment analysis results

[1061] Output: Warning message

[1062] Step 6:

[1063] The server records detailed logs of all operations performed, including timestamps of operations, emotion engine analysis results, fraud pattern detection results, and details of any alert messages sent, for later analysis and troubleshooting.

[1064] Input: Output results of each step

[1065] Output: Detailed log information

[1066] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1067] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1068] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1069] [Fourth embodiment]

[1070] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1071] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1072] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1073] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1074] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1075] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1076] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1077] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1078] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1079] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1080] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1081] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1082] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1083] 1. Collection of audio data

[1084] The device constantly monitors surrounding voice data using a microphone. The device also has a voice recognition sensor that collects user conversations in real time. For example, voice data is acquired when the user is talking on the phone or talking face-to-face with family members.

[1085] 2. Transcription of audio data

[1086] The device uses software to convert the collected voice data into text. This software uses speech recognition technology to analyze the voice data and generate corresponding text. For example, if a user says, "Is this from my bank?", this phrase will be recorded as text.

[1087] 3. Fraud Pattern Detection

[1088] The server receives text sent from the device and uses generative artificial intelligence to detect fraud-related patterns. This AI model is trained on historical fraud datasets and determines in real time whether the user's conversation matches the characteristics of fraud. For example, if a phrase like "Can you tell me my account information?" is detected in the conversation, it will mark it as a possible fraud.

[1089] 4. Sending alerts

[1090] If a fraud-related pattern is detected, the server immediately sends an alert. Recipients of the alert include pre-registered family members and police agencies. For example, if a user utters a typical fraudulent phrase like "Do I need to provide my account number?", the server will send an alert message to family members and the police saying, "A possible fraudulent conversation has been detected."

[1091] Specific examples

[1092] User Status

[1093] The user is an elderly person who regularly communicates with his family and bank over the phone. One day, he receives a call saying, "Hello, is this the bank? I'd like to check my account information."

[1094] Device behavior

[1095] The terminal that receives this conversation immediately collects the voice data and converts it into text information.

[1096] Server Operation

[1097] The user's conversation is sent to a server, which uses generative artificial intelligence to detect fraud patterns. In this case, the phrase "I would like to check my account information" matches the fraud pattern, so it is immediately determined to be a potential fraud.

[1098] Sending alerts

[1099] If a fraudulent conversation is detected, the server sends a warning message to the user's family and the police, saying, "A potential fraudulent conversation has been detected." This allows the family and the police to respond quickly and prevent the user from becoming a victim of fraud.

[1100] The above is a specific embodiment for carrying out the present invention. The present invention protects users from the risk of fraud and allows them to quickly receive support from their families or the police.

[1101] The processing flow will be explained below.

[1102] Specific steps for collecting and processing audio data

[1103] Step 1: Collecting audio data

[1104] The device activates voice recognition software and captures surrounding audio through a microphone, monitoring all user conversations in real time while filtering out background noise and other distracting sounds.

[1105] Step 2: Temporarily save the audio data

[1106] The device temporarily stores the collected voice data in its internal memory. This data is temporarily held for subsequent processing. As voice data is collected, it is accumulated in a buffer.

[1107] Step 3: Transcribe the audio data

[1108] The device uses voice recognition software to convert the temporarily stored voice data into text. The text generated by this process is a textual representation of what the user said. For example, "Hello, is this from my bank?"

[1109] Step 4: Send text information

[1110] The device then sends the converted text data, including necessary metadata (such as a timestamp and user ID), to a server via the Internet.

[1111] Step 5: Detect fraud patterns

[1112] The server analyzes the received text and uses generative artificial intelligence to detect fraud patterns. Specifically, it compares it with historical data sets and identifies phrases and wording characteristic of fraud. This analysis is done in real time.

[1113] Step 6: Determine the likelihood of fraud

[1114] The server evaluates the likelihood of fraud based on the results of the generative AI. If the algorithm determines that there is a high risk of fraud, the result is recorded.

[1115] Step 7: Building the Alert

[1116] If the server detects possible fraud, it constructs an alert message that includes the nature of the detected fraud, the date and time, and the affected user.

[1117] Step 8: Sending an alert

[1118] The server sends alert messages to pre-registered family members and police agencies via email and SMS, allowing recipients to receive the alerts immediately and respond quickly.

[1119] Step 9: Logging

[1120] The server records detailed logs of all operations it performs, including timestamps of operations, fraud patterns detected, and details of warning messages sent, for later analysis and troubleshooting.

[1121] Example 1

[1122] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1123] Today, the risk of becoming a victim of fraud is increasing, especially among the elderly, and there is a need to detect signs of fraud in real time and respond quickly. Furthermore, as fraud methods become more sophisticated, it is difficult to respond with traditional manual monitoring and vigilance, and cooperation from family members and police agencies is essential. Therefore, it is necessary to realize a fraud detection system that uses voice data.

[1124] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1125] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text information, means for transmitting the converted text information to the server, generative artificial intelligence means for detecting fraud-related patterns using the text information received by the server, and means for sending an alert when a fraud-related pattern is detected. This makes it possible to detect signs of fraud in real time, send an alert quickly, and protect users from fraud.

[1126] "Voice data" refers to information that is a digital recording of the user's speech and surrounding sounds.

[1127] "Collection means" refers to a device or method for acquiring audio data using a microphone, sensor, etc.

[1128] "Text information" refers to text data converted from collected voice data using voice recognition technology.

[1129] "Means of converting into textual information" refers to the process of converting audio data into textual data using speech recognition software or algorithms.

[1130] The "means for transmitting the converted character information" is a mechanism for transferring the generated text data to a server via a secure protocol.

[1131] "Generative artificial intelligence" is an AI technology that learns from past data sets and makes predictions and judgments in real time about new data.

[1132] "Fraud-related patterns" refer to specific phrases or patterns of behavior that indicate fraudulent activity.

[1133] "Means for sending an alert" refers to a method for sending a warning message to pre-registered recipients when a fraud-related pattern is detected.

[1134] A "historical fraud dataset" is a collection of previously recorded fraud cases and patterns.

[1135] "Family and Law Enforcement" refers to contacts that include the user's pre-registered relatives and law enforcement agencies.

[1136] MODE FOR CARRYING OUT THE INVENTION

[1137] The present invention relates to a system that uses specific hardware and software to provide real-time fraud protection to users, and a specific embodiment of the system is described below.

[1138] Audio data collection

[1139] The device constantly collects surrounding audio data using a microphone. The device is equipped with a high-performance voice recognition sensor that captures the user's conversation in real time. The device can be a smartphone, tablet, smart speaker, or other device. For example, when a user is talking on the phone, the device can be equipped with a function to record the conversation.

[1140] Transcription of audio data

[1141] The device uses speech recognition software to convert collected voice data into text. Specifically, it uses speech recognition technologies such as the Google Speech-to-Text API and Microsoft Azure Cognitive Services Speech API. This software analyzes the voice data and generates corresponding text. For example, if a user says, "Is this from my bank?", this phrase will be recorded as text on the device.

[1142] Sending audio data

[1143] The device sends the generated text data to the server in an encrypted format using a secure protocol (e.g., HTTPS), which ensures data security and privacy.

[1144] Fraud pattern detection

[1145] The server receives the text sent from the device and begins analyzing it in real time using generative artificial intelligence (e.g., OpenAI GPT). The software on the server is trained on historical fraud datasets and determines whether the text matches the characteristics of fraud. For example, if the phrase "Please tell me your account information" is detected, it will be marked as a possible fraud.

[1146] Sending alerts

[1147] If a fraud-related pattern is detected, the server immediately sends an alert. Recipients of the alert include pre-registered family members and police agencies. For example, if a user says, "Do I need to provide my account number?", the server will send an alert message to family members and the police saying, "A potentially fraudulent conversation has been detected."

[1148] Specific examples

[1149] User Status

[1150] The user is an elderly person who regularly communicates with family and financial institutions over the phone. One day, he receives a call saying, "Hello, is this the bank? I'd like to check my account information."

[1151] Device behavior

[1152] The device that receives this conversation immediately collects the voice data and converts it into text information. The device is a smartphone, and the voice data is converted into text using the Google Speech-to-Text API.

[1153] Server Operation

[1154] The user's conversation is sent to a server, which uses generative artificial intelligence to detect fraud patterns. In this case, the phrase "I would like to check my account information" matches the fraud pattern, so it is immediately determined to be a potential fraud.

[1155] Sending alerts

[1156] If a fraudulent conversation is detected, the server sends a warning message to the user's family and the police, saying, "A potential fraudulent conversation has been detected." This allows the family and the police to respond quickly and prevent the user from becoming a victim of fraud.

[1157] Prompt Sentence Examples

[1158] "If a user calls and says, 'Is this your bank? I'd like to confirm my account information,' we'll mark it as a fraud pattern and send an alert message to their family and the police."

[1159] This explains the configuration of a system for specifically implementing the present invention, which protects users from the risk of fraud and sends prompt alerts to obtain cooperation from family members and the police.

[1160] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1161] Step 1:

[1162] The device constantly collects surrounding audio data using a microphone. When the user speaks, the device captures the audio and records it in digital form. The input is the user's speech, and the output is digital audio data. For example, a user might record themselves saying, "Hello, is this the bank?"

[1163] Step 2:

[1164] The device converts the collected voice data into text using the Google Speech-to-Text API. During this conversion process, voice recognition technology is used to analyze the voice data and generate corresponding text. The input is digital voice data, and the output is text data. Specifically, the voice saying "Hello, is this the bank?" is converted into text "Hello, is this the bank?"

[1165] Step 3:

[1166] The terminal securely transmits the generated text data to the server using the HTTPS protocol. During the data transmission process, data security and privacy are guaranteed by encryption. The input is text data, and the output is secure data transmission to the server. Specifically, the text data "Hello, is this the bank?" is transmitted to the server.

[1167] Step 4:

[1168] The server analyzes the received text data in real time. It runs a generative artificial intelligence (e.g., OpenAI GPT) and compares it with past fraud datasets to detect fraud-related patterns. The input is text data, and the output is the detection result of fraud-related patterns. For example, if the phrase "Please tell me your account information" is detected, it is marked as a possible fraud.

[1169] Step 5:

[1170] If a fraud-related pattern is detected, the server immediately generates an alert and sends it to pre-registered family members and police agencies. During this process, the generated alert message is forwarded to each contact in the distribution list. The input is a fraud-related text phrase, and the output is the alert message sent to family members and police agencies. Specifically, the alert message sent is "A potential fraudulent conversation has been detected."

[1171] Through these steps, the system protects users from the risk of fraud and can quickly issue alerts to prompt action from relevant authorities.

[1172] (Application example 1)

[1173] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1174] In modern society, cases of elderly people falling victim to telephone fraud are increasing. To prevent such fraud, real-time fraud detection and rapid response methods are required. However, conventional methods are insufficient to efficiently monitor telephone conversations and detect possible fraud, making it difficult to completely prevent fraud. Technology to address this issue is needed.

[1175] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1176] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text information, generative artificial intelligence means for detecting fraud-related patterns using the converted text information, means for sending an alert when a fraud-related pattern is detected, means for converting voice data into text using voice recognition software, means for analyzing fraud patterns using a generative artificial intelligence model using a cloud service, and notification means for sending an emergency notification when fraud is detected. This makes it possible to detect potentially fraudulent telephone conversations in real time and quickly notify family members and the police.

[1177] "Audio data" refers to information that records audio in digital format.

[1178] "Means of collection" refers to devices and software for acquiring and storing voice data.

[1179] "Means for converting into textual information" refers to technology or software that converts audio data into text format.

[1180] "Generative artificial intelligence" refers to an AI model that learns from past data and analyzes new data.

[1181] "Detecting fraud-related patterns" is the process of determining whether certain phrases or behaviors are associated with fraud.

[1182] An "alert means" is a system or technology for providing notification in the event of suspected fraud.

[1183] "Speech recognition software" is a program that converts speech into text in real time.

[1184] "Cloud services" are computer resources and data analysis services provided via the Internet.

[1185] A "generative artificial intelligence model" is an AI model used to detect fraud patterns from text data.

[1186] A "notification mechanism" is a system capable of sending emergency notifications based on detected fraud patterns.

[1187] This invention is a real-time fraud detection and emergency notification system that monitors users' phone conversations using their smartphones and can take immediate action if fraudulent activity is suspected.

[1188] System configuration

[1189] The system consists of the following main components:

[1190] 1. User device (smartphone)

[1191] 2. Speech recognition software (Google Speech-to-Text API)

[1192] 3. Cloud Services (Amazon Web Services: AWS Lambda and AWS SNS)

[1193] 4. Generative AI models (e.g., GPT-4)

[1194] Hardware and Software Use

[1195] The user device is equipped with a built-in microphone that is used to collect real-time phone conversations, and this voice data is converted into text using the Google Speech-to-Text API.

[1196] The converted text is sent to AWS Lambda, a cloud service, and analyzed by a generative AI model (e.g., GPT-4), which is trained on historical fraud datasets to detect phrases and patterns characteristic of fraud.

[1197] If a suspicious pattern is detected, AWS SNS will send an emergency notification to pre-registered family members and law enforcement agencies, including a message such as "A potential fraudulent conversation has been detected."

[1198] Specific examples of processing

[1199] User Status

[1200] The user is an elderly person who regularly communicates with family, banks, etc. by telephone. One day, he receives a call saying, "Hello, this is the bank. I would like to confirm my account information."

[1201] Device behavior

[1202] The user's device collects this conversation in real time and converts the audio data into text using the Google Speech-to-Text API.

[1203] Server Operation

[1204] The converted text is sent to AWS Lambda in the cloud and analyzed using a generative artificial intelligence model (e.g., GPT-4). If the detected patterns match the characteristics of a fraud, an emergency notification is sent to family members and police via AWS SNS.

[1205] Prompt Sentence Examples

[1206] Determine whether the following conversation is likely to be a scam and, if so, raise a red flag.

[1207] Conversation: This is a bank. Please let me know your account number so I can process your request.

[1208] Risk of fraud:

[1209] The system protects users from the risk of fraud and allows them to quickly access family or police assistance.

[1210] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1211] Step 1:

[1212] A user terminal collects the telephone conversation.

[1213] Specific operation: When a user starts a call, voice data is collected in real time using the smartphone's built-in microphone.

[1214] Input: Audio data from a user's phone conversation.

[1215] Output: The audio data is saved to the device.

[1216] Step 2:

[1217] The device converts the collected voice data into text information.

[1218] What it does: The audio data is converted to text using the Google Speech-to-Text API.

[1219] Input: The audio data collected in step 1.

[1220] Output: Textual information in text format.

[1221] Step 3:

[1222] The terminal transmits character information in text format to the server.

[1223] Specific operation: The converted text information is uploaded to a server via the Internet.

[1224] Input: The textual information generated in step 2.

[1225] Output: Textual information stored on the server.

[1226] Step 4:

[1227] The server analyzes the text information using a generative artificial intelligence model.

[1228] Specific operation: A generative artificial intelligence model (such as GPT-4) is launched on the server side using AWS Lambda and analyzes text information.

[1229] Input: The textual information sent to the server in step 3.

[1230] Output: Fraud probability assessment results.

[1231] Step 5:

[1232] The server detects fraud patterns.

[1233] How it works: Generative AI models detect phrases and patterns characteristic of fraud.

[1234] Input: The text parsed in step 4.

[1235] Output: Judgment result on whether or not there is suspicion of fraud.

[1236] Step 6:

[1237] If a fraud pattern is detected, the server sends an emergency notification.

[1238] What it does: Uses AWS SNS to send emergency notifications to pre-registered family members and police agencies.

[1239] Input: The probability of fraud determined in step 5.

[1240] Output: Urgent notification message (e.g. "Potential fraudulent conversation detected").

[1241] Step 7:

[1242] Family members and police are notified and begin responding.

[1243] Specific actions: Family members or police who receive the notification will contact the user and take action.

[1244] Input: The emergency notification sent in step 6.

[1245] Output: Ensuring user safety and implementing fraud prevention measures.

[1246] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1247] 1. Collection of audio data

[1248] The device constantly collects surrounding audio data using a microphone. When a user is having a phone call or a face-to-face conversation, the audio data is monitored in real time. This data collection is equipped with a noise filtering function to ensure clear audio data.

[1249] 2. Transcription of audio data

[1250] The device runs speech recognition software to convert the collected voice data into text. The software analyzes the speech and generates corresponding text data. For example, if a user says, "Is this from my bank?", that phrase is recorded verbatim as text.

[1251] 3. Analysis by Emotion Engine

[1252] The device uses an emotion engine to further analyze the converted voice data. The emotion engine analyzes the tone, speed, and phrasing of the user's voice to identify the user's emotional state. For example, if the user says "Are you sure it's safe?" in an anxious voice, the emotion engine will detect anxiety and fear.

[1253] 4. Fraud Pattern Detection

[1254] The server receives text sent from the device and uses generative artificial intelligence to detect fraud-related patterns. The model compares it with historical fraud datasets to identify phrases and behavioral patterns characteristic of fraud. If a user engages in a conversation that includes the phrase "Please tell me my account information," it is determined to be a potential fraud.

[1255] 5. Integrated judgment of emotional state and deception pattern

[1256] The server combines the analysis results from the emotion engine and the fraud pattern detection results from the generative AI to make a comprehensive judgment. If the user's emotional state is biased toward anxiety or fear, and if fraud-related patterns are detected, the server will assess the possibility of fraud as high.

[1257] 6. Building and sending alerts

[1258] If a fraudulent conversation is deemed likely, the server creates a warning message. This message includes the detected fraudulent pattern, the user's emotional state, the date and time of the detection, and the user's information. The warning message is sent to pre-registered family members and police agencies. For example, a message stating "A potentially fraudulent conversation has been detected and the user is in a state of anxiety" is sent.

[1259] 7. Logging

[1260] The server records detailed logs of all operations performed, including timestamps of operations, emotion engine analysis results, fraud pattern detection results, and details of any alert messages sent, for later analysis and troubleshooting.

[1261] Specific examples

[1262] User Status

[1263] The user is an elderly person who regularly communicates with his family and bank over the phone. One day, he received a call saying, "Hello, is this your bank? I'd like to confirm my account information." During the conversation, the user asked in an anxious voice, "Is this really safe?"

[1264] Device behavior

[1265] The device that receives this conversation collects the voice data and converts it into text information. At the same time, the emotion engine analyzes the tone and speed of the user's voice to detect feelings of anxiety.

[1266] Server Operation

[1267] The user's conversational content and emotional state are sent to the server, which uses generative artificial intelligence to detect fraud-related patterns. For example, if the phrase "I want to check my account information" is detected, and the emotion engine indicates that the user is in an anxious state, it will determine that there is a high possibility of fraud.

[1268] Sending an alert

[1269] If it is determined that there is a high possibility of fraud, the server will send a warning message to the family and the police stating that "a potentially fraudulent conversation has been detected and the user is in a state of anxiety."

[1270] The above is a concrete example of how to implement the present invention by combining an emotion engine. This system protects users from the risk of fraud with a higher degree of accuracy and enables quicker response by family members and the police.

[1271] The processing flow will be explained below.

[1272] Specific steps for collecting and analyzing audio data

[1273] Step 1: Collecting audio data

[1274] The device uses a built-in microphone to capture the user's surrounding sounds in real time, recording all conversations whether the user is making a phone call or talking face-to-face. The device also has a filtering function to reduce environmental noise, ensuring clear audio data is collected.

[1275] Step 2: Temporarily save the audio data

[1276] The device temporarily stores the collected voice data in its internal memory. This ensures stable storage of the voice data and makes it available for subsequent processing. Because the voice data is temporarily stored as a buffer, processing can proceed without interruption.

[1277] Step 3: Transcribe the audio data

[1278] The device uses voice recognition software to analyze the temporarily stored voice data and convert it into text data. For example, a user might say, "Hello, is this from my bank?" and the transcription is then adjusted for accuracy.

[1279] Step 4: Send text information

[1280] The device sends the converted text data and associated metadata (such as user ID and timestamp) to the server, which then promptly transmits the data over the internet connection.

[1281] Step 5: Analyze emotional state

[1282] The server analyzes the voice data received from the device using an emotion engine to evaluate the user's emotional state. The emotion engine analyzes the tone, speed, and strength of the voice to detect emotions such as "anxiety" and "fear." For example, if a user says "Is it really safe?" in a tense voice, the emotion of anxiety is detected.

[1283] Step 6: Detect fraud patterns

[1284] The server then inputs the received text data, along with the emotional state, into a generative AI model to detect fraud-related patterns. The AI ​​model then matches it with a large amount of historical data to identify phrases and phrasing associated with fraudulent activity. For example, the phrase "Please tell me your account information" might be detected.

[1285] Step 7: Comprehensive Decision

[1286] The server integrates the analysis results of the emotion engine and the fraud pattern detection results to comprehensively evaluate the risk of fraud. If the user's emotional state is evaluated as "anxiety" or "fear" and also matches the characteristic fraud patterns, it is determined that there is a very high possibility of fraud.

[1287] Step 8: Building the Alert

[1288] If a fraudulent conversation is detected, the server immediately creates an alert message, which includes the details of the detected fraud, the user's emotional state, the date and time, the user ID, etc. For example, the alert might say, "A potentially fraudulent conversation has been detected and the user is in a state of anxiety."

[1289] Step 9: Sending an alert

[1290] The server then sends the constructed warning message to pre-registered family members and police agencies via email and SMS, ensuring that recipients receive the warning quickly.

[1291] Step 10: Logging

[1292] The server records detailed logs of all operations performed, including timestamps of operations, emotion engine and generative AI results, and details of warning messages. This log data can be used for future analysis and troubleshooting.

[1293] Specific examples

[1294] User Status

[1295] The user is an elderly person who regularly has phone and face-to-face conversations. One day, he received a call saying, "Hello, is this your bank? I'd like to confirm my account information." During the conversation, the user said in an anxious voice, "Is this really safe?"

[1296] Device behavior

[1297] The device that captures this conversation collects the voice data, transcribes it, and immediately sends the results to the server. At the same time, the emotion engine detects anxiety from the user's tone of voice.

[1298] Server Operation

[1299] Based on the received text data and the user's emotional state, the server uses generative artificial intelligence to detect fraud patterns. For example, if the phrase "I want to check my account information" matches the characteristics of a fraud, and the user's anxiety is detected, the server determines that there is a high possibility of fraud.

[1300] Sending an alert

[1301] Because it is determined that there is a high possibility of fraud, the server sends a warning message to the family and the police stating that "a potentially fraudulent conversation has been detected and the user is in a state of anxiety."

[1302] This provides users with far greater protection from the risk of fraud and allows for faster response.

[1303] Example 2

[1304] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1305] In modern society, voice-mediated fraud is on the rise, and elderly people and those who are not familiar with information are particularly vulnerable to becoming victims. Conventional fraud detection systems have technologies to recognize fraudulent patterns, but they have limitations in their methods of analyzing the user's emotional state in real time and making an integrated judgment to further increase the likelihood of fraud. This has led to frequent false positives, increasing the possibility that appropriate warnings will not be issued in situations where protection is truly needed.

[1306] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text information, generative artificial intelligence means for detecting fraud-related patterns using the converted text information, emotion analysis means for analyzing the voice data to identify the user's emotional state, and means for integrating the analysis results of the emotion analysis means and the fraud-related pattern detection results to make a comprehensive judgment. This enables highly accurate fraud detection that takes the user's emotional state into consideration, and effectively protects users from the risk of fraud.

[1307] "Audio data" is signal information including the user's and surrounding sounds.

[1308] "Text information" is information in text format obtained by analyzing voice data.

[1309] "Generative AI methods" are AI techniques that learn from past data and generate and identify specific patterns based on newly input data.

[1310] "Emotion analysis means" refers to technology and devices that identify the user's emotional state from voice data and text information.

[1311] "Fraud-related patterns" refer to specific phrases or behavioral characteristics that characterize fraudulent activity.

[1312] "Means for making a comprehensive judgment" refers to the technology and process that integrates the results of sentiment analysis and fraud-related pattern detection to make a final judgment.

[1313] "Means for sending an alert" means the functions and techniques for constructing and sending an alert message to a designated recipient when fraud is determined to be likely.

[1314] The "historical fraud dataset" is a database that collects data on fraud cases that have occurred in the past.

[1315] "Pre-registered family members and police agencies" are emergency contacts that the user has preset in the system.

[1316] This system collects and analyzes user voice data to detect and warn of fraudulent activity early. The following hardware and software are used to implement this system:

[1317] 1. Collection of audio data

[1318] The device uses a microphone to constantly collect sounds around the user. Noise-canceling technology is used to ensure clear audio data. Specifically, a smartphone or a dedicated audio collection device is used.

[1319] 2. Transcription of audio data

[1320] The device runs speech recognition software, such as the Google Speech-to-Text API or Microsoft Azure Cognitive Services, to convert the collected voice data into text. For example, if a user says, "Is this from my bank?", the exact phrase is recorded as text data.

[1321] 3. Analysis by Emotion Engine

[1322] The device runs an emotion engine to analyze the voice data converted into text. The emotion engine analyzes the user's tone of voice, speech rate, intonation, etc. to identify the user's emotional state. Emotion engines used include IBM Watson Tone Analyzer and Lexalytics' Sentiment Analysis API. For example, if a user says "Is this really okay?" in an anxious voice, the emotion engine will detect the "anxiety."

[1323] 4. Fraud Pattern Detection

[1324] The server receives the text sent from the device and uses a generative artificial intelligence model to detect fraud-related patterns. This model compares past fraud datasets. Specifically, OpenAI's GPT-3 and Google BERT are used as generative technologies. If a user engages in a conversation that includes a phrase such as "Please tell me my account information," this is likely to be judged as fraud.

[1325] 5. Integrated judgment of emotional state and deception pattern

[1326] The server combines the emotion analysis results from the emotion engine with the fraud pattern detection results from the generative AI to make a comprehensive judgment. For example, if the emotion analysis result is "anxiety" and the fraud pattern matches, it will assess the possibility of fraud. This information is combined according to a pre-set algorithm by the system.

[1327] 6. Building and sending alerts

[1328] If it is determined that there is a high possibility of fraud, the server will construct an alert message. This message will include the detected fraud pattern, the user's emotional state, the detection date and time, and user information. The alert message will be sent to pre-registered family members and police agencies. In a specific implementation, the alert email will be sent using the SMTP protocol.

[1329] (Example)

[1330] The user is an elderly person who regularly communicates with family and the bank over the phone. One day, he receives a call saying, "Hello, is this the bank? I'd like to check my account information." During the conversation, the user says in an anxious voice, "Is it really safe?" The device that receives this conversation collects the voice data and converts it into text. At the same time, the emotion engine analyzes the tone and speed of the user's voice to detect feelings of anxiety.

[1331] The server receives the user's conversation content and emotional state and uses generative artificial intelligence to detect fraud-related patterns. For example, the phrase "I want to check my account information" is detected, and the emotion engine indicates that the user is in an anxious state. As a result, it is determined that there is a high possibility of fraud, and the server sends an alert message to the user's family and the police stating, "A conversation that may be fraudulent has been detected, and the user is in an anxious state."

[1332] The above is a concrete example of how to implement a fraud detection system that combines an emotion engine. This system will protect users from fraud risks with a higher degree of accuracy and enable quicker response by family members and the police.

[1333] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1334] Step 1:

[1335] The device uses a microphone to collect audio data from the user's surroundings. It then uses the device's built-in noise-canceling technology to capture clear audio data, reducing noise and providing highly accurate audio data. The collected audio data is temporarily stored in the device's memory.

[1336] Input: Ambient audio

[1337] Output: Clear audio data

[1338] Step 2:

[1339] The device sends the collected voice data to the Google Speech-to-Text API and converts it into text. During this process, the voice data is analyzed and corresponding text data is generated. The generated text data is then passed on to the next processing step.

[1340] Input: Clear voice data

[1341] Output: Character information (text data)

[1342] Step 3:

[1343] The device sends the text information to the emotion engine, which uses IBM Watson Tone Analyzer. This engine analyzes the tone, speed, and intonation of the words in the text data to identify the user's emotional state. The analysis results in an emotion label, such as "anxiety" or "joy."

[1344] Input: Character information (text data)

[1345] Output: Sentiment analysis results (emotion labels)

[1346] Step 4:

[1347] The device sends the text information and the results of the emotion analysis to the server. The server inputs the text information into a generative AI model as a prompt sentence. The prompt sentence is a specific sentence based on the text information. The generative AI model (e.g., GPT-3) analyzes the prompt sentence and detects fraud-related patterns. The detection result is a label such as "Fraud pattern detected" or "No problem."

[1348] Input: Text information, emotion analysis results

[1349] Output: Fraud-related pattern detection results

[1350] Step 5:

[1351] The server integrates the results of the sentiment analysis and the fraud-related pattern detection to make a comprehensive judgment. If the sentiment analysis result is "anxiety" and a fraud pattern is detected, the possibility of fraud is assessed as high. The result of this comprehensive judgment is used to construct a warning message.

[1352] Input: Sentiment analysis results, fraud-related pattern detection results

[1353] Output: Overall judgment result

[1354] Step 6:

[1355] The server then creates a warning message based on the comprehensive judgment results, which includes the detected fraud pattern, the user's emotional state, the detection date and time, user information, etc. This warning message is then sent to pre-registered family members and police agencies using the SMTP protocol.

[1356] Input: Overall judgment result

[1357] Output: Warning message

[1358] Step 7:

[1359] The server records detailed logs of all processing steps, including processing timestamps, sentiment analysis results, fraud pattern detection results, and details of any alert messages sent. This log is stored for later analysis and troubleshooting.

[1360] Input: All processing results

[1361] Output: Log data

[1362] (Application example 2)

[1363] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1364] The risk of elderly and socially vulnerable people falling victim to fraud continues to increase. Conventional fraud prevention systems have limitations in their ability to detect fraudulent patterns and do not adequately analyze the user's emotional state, making them unable to completely prevent fraud. There is a need for a system that can resolve these issues, more accurately and quickly identify potential fraud, and send alerts.

[1365] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1366] In this invention, the server includes means for collecting voice data, means for converting the collected voice data into text information, means for analyzing the user's emotional state based on the converted text information, generative artificial intelligence means for detecting fraud-related patterns using the analysis results, and means for sending an alert when a fraud-related pattern is detected. This makes it possible to comprehensively analyze the user's emotional state and fraud-related patterns, and to send an alert quickly and accurately when there is a high possibility of fraud.

[1367] The "means for collecting voice data" refers to a device that has the function of capturing and recording external voice in real time using a microphone on the terminal.

[1368] The "means for converting collected voice data into text information" is a device that has the function of analyzing voice data using voice recognition software and generating corresponding text data.

[1369] The "means for analyzing the user's emotional state based on the converted text information" is a device that has the function of analyzing the user's speaking tone, speed, choice of words, etc., and identifying the user's emotional state using an emotion engine.

[1370] A "generative artificial intelligence means" is an artificial intelligence system that has the ability to learn specific patterns or trends based on past data sets and detect those patterns or trends based on newly input data.

[1371] A "means for detecting fraud-related patterns" is a device capable of using generative artificial intelligence to identify phrases and behavioral patterns characteristic of fraudulent activity.

[1372] The "means for sending an alert" is a device that has the function of automatically sending an alert message to pre-registered recipients when there is a high possibility of fraud.

[1373] The present invention provides a system that collects voice data from a user, analyzes the data to detect the user's emotional state and fraud-related patterns, and sends alerts as needed.

[1374] 1. Collection of audio data

[1375] The device constantly collects voice data using a microphone, allowing it to capture conversations in real time as users interact. The collected voice data is also equipped with a noise filtering function to ensure clear voice data.

[1376] 2. Transcription of audio data

[1377] The collected voice data is converted into text using voice recognition software. For example, Google Speech-to-Text API is commonly used. This software analyzes the voice data and generates corresponding text data. For example, if a user says, "Is this from my bank?", that phrase is accurately recorded as text.

[1378] 3. Analysis by Emotion Engine

[1379] The server then uses an emotion engine to further analyze the converted voice data. The IBM Watson Tone Analyzer is often used as the emotion engine. This engine analyzes the tone, speed, and phrasing of the user's voice to identify the user's emotional state. For example, if the user says "Is it really safe?" in an anxious voice, the emotion engine will detect anxiety.

[1380] 4. Fraud Pattern Detection

[1381] Generative AI is used to detect fraud patterns from the converted text information. Examples of generative AI include OpenAI GPT-3. This method compares with past fraud datasets to identify fraud-related behavioral patterns and phrases. For example, if a conversation includes the phrase "Please tell me your account information," the system will determine that there is a high possibility of fraud.

[1382] 5. Sending an Alert

[1383] If it determines that there is a high possibility of fraud, the server will send a warning message to pre-registered contacts (family or police agencies). This message will include the detected fraud pattern, the user's emotional state, the date and time of detection, and user information. For example, a warning message will be sent stating, "A potentially fraudulent conversation has been detected and the user is in a state of anxiety."

[1384] 6. Logging

[1385] The server records a detailed log of all operations performed, including timestamps of operations, emotion engine analysis results, fraud pattern detection results, details of alert messages sent, etc. This log can be used for later analysis and troubleshooting.

[1386] Specific examples

[1387] For example, suppose an elderly person receives a phone call saying, "Hello, is this the bank? I'd like to check my account information." The device receiving this conversation collects voice data and converts it into text. At the same time, an emotion engine analyzes the tone and speed of the user's voice to detect feelings of anxiety. The content of the user's conversation and emotional state are sent to a server, which uses generative artificial intelligence to detect fraud-related patterns. For example, if the phrase "I'd like to check my account information" is detected and the emotion engine indicates that the user is in an anxious state, it is determined that there is a high possibility of fraud. As a result, the server sends an alert message to the family and police stating, "A potentially fraudulent conversation has been detected and the user is in an anxious state."

[1388] Prompt Sentence Examples

[1389] Determine if the following text contains any signs of fraud: "Hello, is this your bank? I'd like to verify my account information."

[1390] This invention provides users with high accuracy protection from fraud and can send alerts quickly, allowing for a quick response by family members or police.

[1391] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1392] Step 1:

[1393] The device uses a microphone to collect surrounding audio data, which is then saved in real time and filtered to ensure clear audio.

[1394] Input: External audio

[1395] Output: filtered audio data

[1396] Step 2:

[1397] The filtered voice data is converted into text using voice recognition software (e.g., Google Speech-to-Text API).

[1398] Input: filtered audio data

[1399] Output: Character information (text data)

[1400] Step 3:

[1401] The text information is sent to a server, and the user's emotional state is analyzed using an emotion engine (e.g., IBM Watson Tone Analyzer). The emotion engine analyzes the tone, speed, and phrasing of the text data to identify the user's emotion.

[1402] Input: Character information (text data)

[1403] Output: Emotion analysis results (anxiety, anger, joy, etc.)

[1404] Step 4:

[1405] The server uses generative AI (e.g., OpenAI GPT-3) to detect fraud-related patterns based on the sentiment analysis results and text information. This AI compares the results with fraud patterns learned from past datasets to find signs of fraud.

[1406] Input: Emotion analysis results, character information (text data)

[1407] Output: Fraud pattern detection results

[1408] Step 5:

[1409] If a fraudulent activity is detected, the server sends a warning message to pre-registered contacts, which includes information about the detected fraudulent activity, the user's emotional state, the date and time of the detection, and other user information.

[1410] Input: Fraud pattern detection results, sentiment analysis results

[1411] Output: Warning message

[1412] Step 6:

[1413] The server records detailed logs of all operations performed, including timestamps of operations, emotion engine analysis results, fraud pattern detection results, and details of any alert messages sent, for later analysis and troubleshooting.

[1414] Input: Output results of each step

[1415] Output: Detailed log information

[1416] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1417] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1418] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1419] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1420] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1421] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1422] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1423] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1424] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1425] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1426] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1427] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1428] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1429] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1430] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1431] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1432] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1433] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1434] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1435] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1436] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1437] The following is further disclosed regarding the above embodiment.

[1438] (Claim 1)

[1439] means for collecting audio data;

[1440] A means for converting the collected voice data into text information;

[1441] a generative artificial intelligence means for detecting fraud-related patterns using the converted text information;

[1442] means for sending an alert when a fraud-related pattern is detected;

[1443] A system including:

[1444] (Claim 2)

[1445] 10. The system of claim 1, further comprising means for matching against historical fraud data sets in detecting fraud-related patterns.

[1446] (Claim 3)

[1447] 10. The system of claim 1, further comprising means for sending the alert to pre-registered family and law enforcement contacts as recipients of the alert.

[1448] "Example 1"

[1449] (Claim 1)

[1450] means for collecting audio data;

[1451] A means for converting the collected voice data into text information;

[1452] means for transmitting the converted character information to a server;

[1453] a generative artificial intelligence means for detecting fraud-related patterns using the character information received by the server;

[1454] means for sending an alert when a fraud-related pattern is detected;

[1455] A system including:

[1456] (Claim 2)

[1457] 10. The system of claim 1, further comprising generative artificial intelligence means for matching against historical fraud data sets in detecting fraud-related patterns.

[1458] (Claim 3)

[1459] 10. The system of claim 1, further comprising means for sending the alert to pre-registered family and law enforcement contacts as recipients of the alert.

[1460] "Application Example 1"

[1461] (Claim 1)

[1462] means for collecting audio data;

[1463] A means for converting the collected voice data into text information;

[1464] a generative artificial intelligence means for detecting fraud-related patterns using the converted text information;

[1465] means for sending an alert when a fraud-related pattern is detected;

[1466] a means for converting the voice data into text using speech recognition software;

[1467] a means for analyzing fraud patterns using a generative artificial intelligence model using a cloud service;

[1468] notification means for sending urgent notifications if fraud is detected;

[1469] A system including:

[1470] (Claim 2)

[1471] 10. The system of claim 1, further comprising means for matching against historical fraud data sets in detecting fraud-related patterns.

[1472] (Claim 3)

[1473] 10. The system of claim 1, further comprising means for sending the alert to pre-registered family and law enforcement contacts as recipients of the alert.

[1474] 10. The system of claim 1, utilizing a generative artificial intelligence model for fraud pattern detection using cloud services.

[1475] "Example 2: Combining Emotion Engines"

[1476] (Claim 1)

[1477] means for collecting audio data;

[1478] A means for converting the collected voice data into text information;

[1479] a generative artificial intelligence means for detecting fraud-related patterns using the converted text information;

[1480] means for sending an alert when a fraud-related pattern is detected;

[1481] emotion analysis means for analyzing the voice data to identify the emotional state of the user;

[1482] a means for integrating the analysis results of the emotion analysis means and the fraud-related pattern detection results to make a comprehensive judgment;

[1483] A system including:

[1484] (Claim 2)

[1485] 10. The system of claim 1, further comprising means for matching against historical fraud data sets in detecting fraud-related patterns.

[1486] (Claim 3)

[1487] 10. The system of claim 1, further comprising means for sending the alert to pre-registered family and law enforcement contacts as recipients of the alert.

[1488] "Application example 2 when combining emotion engines"

[1489] (Claim 1)

[1490] means for collecting audio data;

[1491] A means for converting the collected voice data into text information;

[1492] A means for analyzing the emotional state of the user based on the converted character information;

[1493] a generative artificial intelligence means for detecting fraud-related patterns using the analysis results; and

[1494] means for sending an alert when a fraud-related pattern is detected;

[1495] A system including:

[1496] (Claim 2)

[1497] 10. The system of claim 1, further comprising: means for analyzing the user's emotional state based on the converted text information; and means for evaluating voice tone and rate to analyze the emotional state.

[1498] (Claim 3)

[1499] 10. The system of claim 1, further comprising means for sending the alert to a contact that has been pre-registered as a recipient of the alert. [Explanation of symbols]

[1500] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for collecting audio data; A means for converting the collected voice data into text information; a generative artificial intelligence means for detecting fraud-related patterns using the converted text information; means for sending an alert when a fraud-related pattern is detected; A system including:

2. The system of claim 1 , further comprising means for matching against historical fraud data sets in detecting fraud-related patterns.

3. 10. The system of claim 1, further comprising means for sending the alert to pre-registered family and law enforcement contacts as recipients of the alert.

Images