system

Abstract

A system is provided.SOLUTION: A system comprising: means for collecting voice data during a call in real-time and converting the voice to text format; means for analyzing the converted text data and detecting patterns of special fraud; means for sending an alert notification to a set contact if the likelihood of fraud is high; means for blocking the call if the likelihood of fraud is determined to be high; and means for sending a warning message to the appropriate ATM.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The number of victims of special frauds is increasing year by year, causing many people not only financial losses but also great psychological damage. Telephone fraud, in particular, is particularly prone to deceive victims, as fraudsters use sophisticated persuasive techniques. To prevent such fraudulent activities, it is essential to detect signs of fraud early and respond quickly. However, currently, there are no adequate means established for identifying fraud and preventing damage, making it difficult to intervene before victims suffer harm. The present invention aims to solve this problem and provide a system that prevents fraud damage. [Means for solving the problem]

[0005] The present invention includes a means for collecting voice data during a call in real time and converting the voice into text format. It also includes a means for analyzing the converted text data and detecting patterns of special fraud. It also provides a means for sending an alert notification to a set contact if a fraudulent call is determined to be highly likely. It also includes a means for blocking the call if a fraudulent call is highly likely. It also includes a means for sending a warning message to the relevant ATM, and the ATM that receives the warning message displays a warning and implements additional restrictions according to specific conditions. In this way, it is possible to detect telephone fraud early and prevent fraud damage by implementing multi-stage responses before victims fall victim to the fraud.

[0006] "In-call voice data" refers to voice information collected through a microphone during a telephone communication, and is data represented in digital form.

[0007] "Real-time collection" refers to a process in which voice data is collected continuously without delay while a call is being made.

[0008] A "speech-to-text converter" is a device or method that uses speech recognition technology to convert audio information into corresponding text.

[0009] "Converted text data" is information of a character string generated from voice data using voice recognition technology.

[0010] "Analyzing" is the process of analyzing text data and understanding its content and patterns.

[0011] "Special fraud patterns" are characteristic keywords, phrases, or contexts that indicate fraudulent activity and are signs of fraud.

[0012] A "detecting means" is a device or method that uses an algorithm or program to find a particular pattern.

[0013] "High probability of fraud" means that the probability of fraud exceeds a certain threshold based on the analysis results.

[0014] "Set contacts" refers to contact information of family members or related parties who have been registered in advance, and to whom notifications will be sent.

[0015] An "alert notification" is a warning message sent to notify you that fraudulent activity has been detected.

[0016] A "call disconnection means" is a device or method for terminating an ongoing call.

[0017] A "relevant ATM" is an automated teller machine that a victim may use.

[0018] A "warning message" is a notification sent to alert you to the presence of a particular danger.

[0019] "ATM that received the warning message" is an automated teller machine that received the warning message.

[0020] "Specific conditions" are the criteria established to trigger warnings and additional measures.

[0021] "Additional restrictions" are restrictions on operations and transactions that can be performed at ATMs to prevent potential harm. [Brief explanation of the drawings]

[0022] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0023] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0024] First, the terms used in the following description will be explained.

[0025] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0026] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0027] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0028] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0029] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0030] [First embodiment]

[0031] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0032] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0033] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0034] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0035] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0036] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0037] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0038] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0039] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0041] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0043] The present invention is a system that collects voice data during a call in real time, detects signs of special fraud, and prevents fraud damage before it occurs. The specific processing flow and details of the program for this system are explained below.

[0044] 1. Collecting and converting audio data into text

[0045] Device:

[0046] When a call is initiated, the device collects real-time audio data during the call. Specifically, it uses the device's microphone to record the audio and stores it digitally. The collected audio data is immediately sent to a speech recognition engine, which converts the audio into corresponding text data. This conversion process is carried out using highly accurate speech recognition technology.

[0047] 2. Text Data Analysis

[0048] server:

[0049] The converted text data is sent to a server. The server receives the text data and analyzes it using natural language processing (NLP) techniques. This analysis includes algorithms that detect keywords and phrases characteristic of fraud. Based on the analysis results, it determines whether a fraud pattern exists.

[0050] 3. Sending alert notifications

[0051] server:

[0052] If the analysis determines that a call is likely to be fraudulent, the server immediately sends an alert notification to configured contacts (usually the victim's family or trusted individuals) warning them that a suspected fraudulent call has been detected and recommending actions to take to prevent it.

[0053] 4. Call blocking

[0054] Device:

[0055] If a fraudulent call is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the victim from further interaction with the fraudster.

[0056] 5. ATM warning messages

[0057] server:

[0058] If fraud is likely, the server will send a warning message to ATM networks that the victim may use, stating that a fraud attempt is suspected and urging caution.

[0059] ATM:

[0060] When an ATM receives a warning message, it will display a warning to the user and, if certain conditions are met (e.g., large withdrawals), will trigger additional restrictions, including withdrawal limits and additional identity verification procedures.

[0061] Specific examples

[0062] Example 1: Victim receives a call from a scammer

[0063] 1. Device: The victim's mobile phone receives a call from a scammer. The device collects the audio during the call and converts it into text in real time.

[0064] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[0065] 3. Server: Determines that the call is likely fraudulent and sends an alert to the victim's family, informing them that a suspected fraudulent call has been detected and instructing them to take immediate action.

[0066] 4. Device: Call blocking is implemented, preventing the victim from continuing the call with the scammer.

[0067] 5. Server: Sends a warning message to ATMs that the victim may use, indicating that there is a high possibility of fraud.

[0068] 6. ATMs: ATMs display warning messages to alert users to potential fraud and trigger additional restrictions based on certain conditions.

[0069] In this way, the system of the present invention can detect fraudulent activity in real time during a call and take multi-stage measures to prevent fraud damage before it occurs.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] Device:

[0073] When a call is initiated, the device immediately collects audio data during the call, which is captured through the device's microphone and converted into a digital format.

[0074] Step 2:

[0075] Device:

[0076] The collected voice data is converted into text format in real time. A voice recognition engine is activated, processing the voice data at high speed to generate text data.

[0077] Step 3:

[0078] Device:

[0079] The converted text data is sent to the server, where it is encrypted and transferred using a secure communication protocol.

[0080] Step 4:

[0081] server:

[0082] The server receives the text data and analyzes it using a natural language processing (NLP) engine, which includes detecting keywords and phrases that may indicate fraud.

[0083] Step 5:

[0084] server:

[0085] Based on the analysis results, it is determined whether there is a possibility of fraud. If the probability of fraud exceeds a certain threshold, it is determined that there is a high possibility of fraud.

[0086] Step 6:

[0087] server:

[0088] If a fraudulent call is detected, an alert notification will be sent to your configured contacts, containing a message informing you that a suspected fraudulent call has been detected.

[0089] Step 7:

[0090] server:

[0091] After sending the alert notification, the server sends a command to the device to disconnect the call, which is also encrypted and communicated securely.

[0092] Step 8:

[0093] Device:

[0094] The device receives instructions from the server and automatically ends the current call, preventing the victim from continuing the call with the scammer.

[0095] Step 9:

[0096] server:

[0097] Assuming there is a high probability of fraud, the server will send a warning message to the affected ATM, which will prevent the victim from immediately withdrawing cash.

[0098] Step 10:

[0099] ATM:

[0100] When an ATM receives a warning message, it will display a warning accordingly, and if the user meets certain conditions (for example, if they attempt to withdraw a large amount), additional restrictions will be automatically implemented.

[0101] Example 1

[0102] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0103] Currently, the number of victims of special frauds is on the rise, and the methods targeting the elderly in particular are becoming more sophisticated. An effective system to prevent such fraud is needed, but existing systems have difficulty responding in real time, and countermeasures are often only implemented after a crime has occurred. For this reason, there is an urgent need to develop a system that can detect signs of fraud in real time during a call and respond immediately.

[0104] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0105] In this invention, the server includes means for collecting voice data during a call in real time and converting the voice into text format, means for analyzing the converted text data using natural language processing technology to detect patterns of special fraud, and means for creating prompt sentences using a generative model to evaluate the possibility of fraud. This makes it possible to detect signs of special fraud in real time during a call and immediately send an alert or block the call.

[0106] "During a call, voice data" refers to digital data of voice collected using a microphone while a user is making a call.

[0107] "Means for converting to text" refers to the speech recognition technology or software used to convert collected voice data into text.

[0108] "Natural language processing technology" is an artificial intelligence technology that analyzes text data, extracts information through an understanding of vocabulary, grammar, and meaning, and recognizes specific patterns.

[0109] "Special fraud patterns" are a collection of typical phrases, keywords, and behavioral patterns used when committing fraud.

[0110] "Means for sending alert notifications" means communications methods or technologies for sending real-time warning messages to designated contacts in the event of a high likelihood of fraud.

[0111] "Call blocking measures" are technologies or devices that forcibly terminate an ongoing call when it is determined that there is a high possibility of fraud.

[0112] "Means for sending warning messages" refers to technology for sending warning messages to automated teller machine networks that victims may use when signs of fraud are detected.

[0113] "Means for activating additional restrictions" means a function or device for imposing withdrawal limits on transactions or implementing additional identity verification procedures in accordance with certain conditions.

[0114] "Means for creating prompt sentences to assess the likelihood of fraud using a generative model" refers to a technology or method that uses an artificial intelligence model to assess the likelihood of fraud based on the frequency of occurrence of specific phrases or keywords in text data, and generates prompt sentences to present the results to users or administrators.

[0115] The present invention is a system for collecting voice data during a call in real time, detecting signs of special fraud, and preventing fraud damage before it occurs. The system of the present invention is mainly composed of terminals, a server, and an ATM.

[0116] Collecting and converting audio data into text

[0117] Device:

[0118] When a call is initiated, the device collects real-time audio data during the call. Specifically, it uses the device's microphone to record the audio and stores it digitally. The collected audio data is immediately sent to a speech recognition engine, which converts the audio into corresponding text data. This conversion process is carried out using high-precision speech recognition technology, such as Google Cloud Speech-to-Text.

[0119] Text data analysis

[0120] server:

[0121] The converted text data is sent to a server. The server receives the text data and analyzes it using natural language processing (NLP) techniques. This analysis includes generative AI models (e.g., BERT or GPT-3) to detect keywords and phrases characteristic of fraud. Based on the analysis results, it determines whether a fraud pattern exists.

[0122] Sending alert notifications

[0123] server:

[0124] If the analysis determines that a call is likely to be fraudulent, the server immediately sends an alert notification to configured contacts (usually the victim's family or trusted individuals) warning them that a suspected fraudulent call has been detected and recommending actions to take to prevent it.

[0125] Call blocking

[0126] Device:

[0127] If fraud is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the user from further interaction with the fraudster.

[0128] ATM warning message

[0129] server:

[0130] If fraud is likely, the server will send a warning message to ATM networks that the victim may use, stating that a fraud attempt is suspected and urging caution.

[0131] ATM:

[0132] When an ATM receives a warning message, it will display a warning to the user and, if certain conditions are met (e.g., large withdrawals), will trigger additional restrictions, including withdrawal limits and additional identity verification procedures.

[0133] Specific examples

[0134] Example 1: Victim receives a call from a scammer

[0135] 1. Device: Imagine a scenario where a victim's mobile phone receives a call from a scammer. The device collects the audio during the call and converts it into text in real time.

[0136] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[0137] 3. Server: Determines that the call is likely fraudulent and sends an alert notification to the victim's family, informing them that a suspected fraudulent call has been detected and providing instructions for immediate action.

[0138] 4. Device: Call blocking is performed, preventing the victim from continuing the conversation with the scammer.

[0139] 5. Server: Determines that there is a high possibility of fraud and sends a warning message to ATMs that the victim may use.

[0140] 6. ATMs: ATMs will display warning messages to notify users of potential fraud, and additional restrictions will be implemented if certain conditions are met.

[0141] Prompt Sentence Examples

[0142] Prompt: "Decide if this call is a scam"

[0143] Generative AI models analyze the frequency of specific keywords and phrases to assess the likelihood of fraud and, based on the results, recommend countermeasures in cases where fraud is suspected.

[0144] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0145] Step 1:

[0146] Start of voice data collection

[0147] Device:

[0148] When a user starts a call, the device detects the start of the call. At this time, the device's microphone becomes active and collects audio data during the call in real time. The collected audio data is saved in digital format. The input is the user's voice during the call, and the output is the collected digital audio data. Specifically, when the user presses the answer button, the device detects the start of the call using an event listener.

[0149] Step 2:

[0150] Converting audio data to text

[0151] Device:

[0152] The device sends the collected voice data to a voice recognition engine, which (for example, Google Cloud Speech-to-Text API) converts the voice into text data in real time. The input is digital voice data, and the output is the corresponding text data. Specifically, the voice recorded by the microphone is stored in digital format and sent to the voice recognition engine.

[0153] Step 3:

[0154] Sending text data

[0155] Device:

[0156] The converted text data is sent to the server. The input is text data, and the output is data transmission to the server. Specifically, the terminal sends the text data to the server using an HTTP request.

[0157] Step 4:

[0158] Text data analysis

[0159] server:

[0160] The server analyzes the received text data using natural language processing technology. This analysis uses a generative AI model that detects keywords and phrases characteristic of fraud. The input is text data, and the output is the analysis result (whether or not there is a possibility of fraud). Specifically, the server analyzes the text data using a generative AI model such as BERT or GPT-3.

[0161] Step 5:

[0162] Assessment of the likelihood of fraud

[0163] server:

[0164] The server uses a generative AI model to assess the likelihood of fraud and create a prompt. The input is analyzed text data, and the output is a prompt that assesses the likelihood of fraud. Specifically, the generative AI model evaluates specific keywords and phrases and generates a prompt based on the results.

[0165] Step 6:

[0166] Sending alert notifications

[0167] server:

[0168] If the server determines that there is a high possibility of fraud, it will send an alert notification to the configured contacts. The input is the prompt text and the alert notification destination information, and the output is sending the alert notification. Specifically, the server will send the notification using SMS or email API.

[0169] Step 7:

[0170] Call blocking

[0171] Device:

[0172] Based on instructions from the server, the terminal automatically disconnects the current call. The input is a call disconnect command from the server, and the output is the termination of the call. Specifically, the terminal uses the telephone communication module to forcibly terminate the call.

[0173] Step 8:

[0174] Sending warning messages to ATMs

[0175] server:

[0176] If the server determines that there is a high possibility of fraud, it sends a warning message to the ATM network that the victim may use. The input is a warning message and ATM network information, and the output is the transmission of the warning message. Specifically, the server sends a message to the ATM network using a network protocol.

[0177] Step 9:

[0178] ATM warning signs and restrictions

[0179] ATM:

[0180] When an ATM receives a warning message, it displays the warning to the user. Furthermore, if certain conditions are met (e.g., withdrawing a large amount), additional restrictions are implemented. The input is the received warning message and transaction conditions, and the output is the display of the warning and the implementation of restrictions. Specifically, the ATM displays the warning message on its display and, if necessary, performs additional identity verification procedures.

[0181] (Application example 1)

[0182] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0183] In recent years, there has been an increase in specialized frauds using sophisticated methods, and many people have fallen victim to them. In particular, frauds committed over the phone often target the elderly and those with little financial knowledge, so fast and effective countermeasures are needed. However, current systems lack real-time fraud detection capabilities, making it difficult to prevent damage before it occurs. For this reason, there is a need for the development of a system that can quickly detect possible fraud during a call and take appropriate countermeasures.

[0184] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0185] In this invention, the server includes means for collecting voice data during a call in real time and converting the voice into text format, means for analyzing the converted text data and detecting patterns of special fraud, means for sending an alert notification to a set contact if there is a high possibility of fraud, means for blocking the call if it is determined that there is a high possibility of fraud, means for sending a warning message to the relevant ATM, means for displaying the warning message on the user's mobile device, and means for analyzing the text data using a generative AI model. This makes it possible to detect special frauds occurring during a call in real time and take prompt measures.

[0186] "Voice data" refers to digital data of voice collected during a call using a microphone or the like of a terminal.

[0187] "Text format" is a digital data format in which voice data is converted into a string of characters.

[0188] "Converted text data" refers to data obtained by converting voice data into text format using voice recognition technology.

[0189] "Patterns of special fraud" are characteristic patterns in text data that contain keywords, phrases, and structures specific to fraudulent activity.

[0190] An "alert notification" is a warning message sent to configured contacts when potential fraud is detected.

[0191] "Call blocking measures" are features that automatically terminate an active call if there is a high likelihood of fraud.

[0192] A "warning message" is a message that warns of possible fraud.

[0193] An "ATM (Automated Teller Machine)" is an automated machine for conducting financial transactions.

[0194] "Means for displaying a warning message" refers to a function that visually displays a warning message on an ATM or a user's mobile device.

[0195] A "generative AI model" is an artificial intelligence model trained to analyze text data using natural language processing and detect fraudulent patterns.

[0196] The present invention is a system for collecting voice data during a call in real time, detecting signs of special fraud, and preventing fraud damage before it occurs. Detailed embodiments of this system are described below.

[0197] Collecting and converting audio data into text

[0198] Device:

[0199] When a call is initiated, the device uses the smartphone's microphone to collect real-time audio data during the call and stores it digitally. This audio data is then immediately sent to a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the audio into text data.

[0200] Text data analysis

[0201] server:

[0202] The converted text data is then sent to a cloud server, which receives the data and analyzes it using natural language processing (NLP) techniques, using a generative AI model (e.g., OpenAI's GPT-4) to detect keywords and phrases characteristic of fraud.

[0203] Sending alert notifications

[0204] server:

[0205] If the analysis determines that there is a high possibility of fraud, the server immediately sends an alert notification to the user's configured contacts (usually family members or trusted individuals) that warns of the suspected fraud and provides advice on how to respond.

[0206] Call blocking

[0207] Device:

[0208] If a fraudulent call is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the user from further interaction with the fraudster.

[0209] Local Warning Display

[0210] Device:

[0211] After the call is blocked, the user will receive a warning message on their mobile device, including a message about suspected fraud and next steps to take (e.g., contacting police or suspending their card).

[0212] Sending warning messages to ATMs

[0213] server:

[0214] If fraud is likely, the server will send a warning message to ATM networks that the victim may use, stating that a fraud attempt is suspected and urging caution.

[0215] ATM:

[0216] When an ATM receives a warning message, it will display a warning to the user and, if certain conditions are met (e.g., large withdrawals), will trigger additional restrictions, including withdrawal limits and additional identity verification procedures.

[0217] Specific examples

[0218] User Scenarios

[0219] 1. Call Initiation: The user receives a call from an unknown number.

[0220] 2. Speech recognition: During a call, the smartphone app converts voice into text data.

[0221] 3. NLP analysis: The text data is sent to a cloud server and analyzed using the GPT-4 model.

[0222] 4. Alerts: Keyword such as "bank account" or "transfer" are detected and an alert is sent to family members. If necessary, the call will be blocked.

[0223] 5. Local Alert: After the call ends, the user will be shown a warning message and guided on next steps.

[0224] Prompt Sentence Examples

[0225] "The text data of a suspected fraudulent call is shown below. Analyze this text data and determine whether it is likely to be fraudulent. If you are suspicious, output a warning message. Text data: 'A family member is in the hospital in an emergency. I need money immediately, so please transfer it to my bank account.'"

[0226] In this way, the present invention is a system that can detect fraudulent acts during calls in real time and prevent damage by taking multi-stage measures.

[0227] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0228] Step 1:

[0229] Collecting and converting audio data into text

[0230] Device:

[0231] When a call is initiated, the device uses a microphone to collect voice data in real time, stores the voice data in digital format, and immediately sends it to a voice recognition engine (e.g., Google Cloud Speech-to-Text API). Voice data is input and corresponding text data is output. At this time, the voice data is temporarily stored in the device's storage.

[0232] Step 2:

[0233] Sending and analyzing text data

[0234] server:

[0235] The server receives the text data sent from the device. The server sends this text data to a natural language processing (NLP) engine (e.g., OpenAI's GPT-4) to detect characteristic fraud keywords and phrases. The server receives the text data as input and outputs an analysis result indicating the likelihood of fraud.

[0236] Step 3:

[0237] Sending alert notifications

[0238] server:

[0239] If the analysis result indicates a high probability of fraud, the server will send an alert notification to the configured contacts. The alert notification will include information about the suspected fraud and the user's recommended actions. The server takes the analysis result as input and sends an alert notification to the contacts as output.

[0240] Step 4:

[0241] Call blocking

[0242] Device:

[0243] When the device receives a call-hangup command from the server, it automatically hangs up the current call, preventing the user from continuing to interact with the scammer. It takes a command from the server as input and ends the call as output.

[0244] Step 5:

[0245] Local Warning Display

[0246] Device:

[0247] After the call is blocked, the device displays a warning message on the user's screen, informing them of the suspected fraud and providing next steps. The input is the completion of the call blocking, and the output is to display a warning message on the device.

[0248] Step 6:

[0249] Sending warning messages to ATMs

[0250] server:

[0251] If the server determines that there is a particularly high possibility of fraud, it sends a warning message to the ATM network, stating that the user should exercise caution. The server takes as input data indicating a high possibility of fraud, and outputs by sending a warning message to the ATM network.

[0252] Step 7:

[0253] ATM warning message display and restriction measures

[0254] ATM:

[0255] When an ATM receives a warning message, it displays a visual warning to the user. If certain conditions are met, such as a large withdrawal, additional withdrawal restrictions or identity verification procedures are implemented. The input is a warning message from the server, and the output is the display of a warning message or the implementation of a restriction.

[0256] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0257] This invention is a system that collects voice data during calls in real time, detects signs of special fraud, and analyzes the user's emotions to prevent fraud damage before it occurs. The specific processing flow and details of the program for this system are explained below.

[0258] 1. Collecting and converting audio data into text

[0259] Device:

[0260] When a call is initiated, the device immediately collects voice data during the call. The voice data is picked up through the device's microphone and converted into digital form. The collected voice data is immediately sent to a speech recognition engine, which converts the voice into corresponding text data. This conversion process is carried out using highly accurate voice recognition technology.

[0261] 2. Text Data Analysis

[0262] server:

[0263] The converted text data is sent to a server. The server receives the text data and analyzes it using natural language processing (NLP) techniques. This analysis includes algorithms that detect keywords and phrases characteristic of fraud. Based on the analysis results, it determines whether a fraud pattern exists.

[0264] 3. Emotion analysis

[0265] Device:

[0266] Voice data during a call is also sent to the emotion engine, which recognizes emotions from the user's voice. If emotions such as alertness or anxiety are detected, that information is also sent to the server.

[0267] 4. Enhanced alert notifications

[0268] server:

[0269] If the analysis determines that a scam is likely, the results of the sentiment analysis are also taken into account. In particular, if the user's sentiment indicates vigilance or anxiety, the alert notification will be enhanced and a message urging immediate action will be generated. The server will then send the alert notification to the designated contacts (usually the victim's family or trusted individuals).

[0270] 5. Call blocking

[0271] Device:

[0272] If a fraudulent call is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the victim from further interaction with the fraudster.

[0273] 6. ATM warning messages

[0274] server:

[0275] If fraud is likely, the server will send a warning message to ATM networks that the victim may use, stating that a fraud attempt is suspected and urging caution.

[0276] ATM:

[0277] When an ATM receives a warning message, it will display a warning to the user and, if certain conditions are met (e.g., large withdrawals), will trigger additional restrictions, including withdrawal limits and additional identity verification procedures.

[0278] Specific examples

[0279] Example 1: Victim receives a call from a scammer

[0280] 1. Device: The victim's mobile phone receives a call from a scammer. The device collects the audio during the call and converts it into text in real time.

[0281] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[0282] 3. Device: At the same time, the emotion engine analyzes the user's emotions from the voice data during the call. If alertness or anxiety is detected, that information is also sent to the server.

[0283] 4. Server: Determines the likelihood of fraud and takes into account sentiment analysis results to create an enhanced alert, which includes information that a suspected fraudulent call has been detected and instructions to take immediate action.

[0284] 5. Device: Call blocking is implemented, preventing the victim from continuing the call with the scammer.

[0285] 6. Server: Sends a warning message to ATMs that the victim may use, indicating a high probability of fraud.

[0286] 7. ATMs: ATMs display warning messages to alert users to potential fraud and trigger additional restrictions based on certain conditions.

[0287] In this way, the system of the present invention detects fraudulent behavior in real time during a call and takes multi-stage measures by combining it with emotion analysis, thereby making it possible to prevent fraud damage before it occurs.

[0288] The processing flow will be explained below.

[0289] Step 1:

[0290] Device:

[0291] When a call is initiated, the device immediately begins collecting audio data during the call. It uses the device's microphone to capture audio and stores it digitally. It is set to collect audio data continuously.

[0292] Step 2:

[0293] Device:

[0294] The collected voice data is sent to a voice recognition engine in real time and instantly converted into text. The voice recognition engine uses a highly accurate recognition algorithm to generate text data from the voice. The generated text data is then stored in memory.

[0295] Step 3:

[0296] Device:

[0297] The converted text data is encrypted and sent to a server using a secure communication protocol, where it is transferred in real time to the server for analysis.

[0298] Step 4:

[0299] server:

[0300] The server receives the text data, which is then analyzed using a natural language processing (NLP) engine to detect keywords and phrases that indicate fraud.

[0301] Step 5:

[0302] server:

[0303] Based on the analysis results, we assess whether there is a possibility of fraud, using pre-defined criteria and pattern matching algorithms to determine whether there is a high probability of fraud.

[0304] Step 6:

[0305] Device:

[0306] Voice data during a call is simultaneously sent to the emotion engine, which analyzes the user's voice tone, tempo, and other factors to recognize their emotions. Emotion analysis results are generated in real time.

[0307] Step 7:

[0308] Device:

[0309] If the emotion analysis results indicate alarm or anxiety, the results are also encrypted and sent to the server. The emotion information is used as part of the fraud detection algorithm.

[0310] Step 8:

[0311] server:

[0312] If a fraudulent call is deemed likely, the results of the sentiment analysis are taken into account to enhance the alert notification, which includes a message informing users that a suspected fraudulent call has been detected and recommending immediate action.

[0313] Step 9:

[0314] server:

[0315] An alert notification is sent to configured contacts, typically family members or trusted associates of the victim, and the notification is sent immediately.

[0316] Step 10:

[0317] Device:

[0318] It receives call blocking instructions from the server and automatically ends the current call, preventing the victim from having any further contact with the scammer.

[0319] Step 11:

[0320] server:

[0321] If there is a high possibility of fraud, the server sends a warning message to the ATM, urging caution due to suspected fraud.

[0322] Step 12:

[0323] ATM:

[0324] When an ATM receives a warning message, it will display a warning accordingly. If the user meets certain conditions (e.g., withdrawing a large amount), additional restrictions will be triggered. In addition, withdrawal restrictions and additional identity verification procedures will be implemented.

[0325] In this way, the system of the present invention can detect fraudulent activity in real time during a call and take multi-stage measures to prevent fraud damage before it occurs. Furthermore, by analyzing the user's emotions, it can provide more accurate alerts and promote a quicker response.

[0326] Example 2

[0327] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0328] In today's society, where fraud damage is on the rise, there is a growing need for a system that can detect special frauds that occur during phone calls in real time and combine them with user emotion analysis to quickly and accurately prevent fraud. Conventional methods have made it difficult to detect signs of fraud in real time, making it difficult to prevent damage before it occurs. In addition, there is a demand for more accurate fraud detection by incorporating user emotion analysis.

[0329] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0330] In this invention, the server includes means for collecting voice data during a call in real time and converting the voice into text format, means for analyzing the converted text data and detecting patterns of special fraud, and means for analyzing emotions from the collected voice data in real time. This makes it possible to detect signs of special fraud occurring during a call in real time and to respond quickly and accurately based on the analysis of the user's emotions.

[0331] "Voice data during a call" refers to digital data of the voice collected through the microphone of the terminal while a call is being made.

[0332] "Real-time collection" refers to the process of capturing audio data on the fly without delay.

[0333] The "means for converting voice into text format" is a mechanism for converting voice data into corresponding character string data using voice recognition technology.

[0334] "Converted text data" is character string data generated by speech recognition technology.

[0335] "Analysis" is the process of extracting and evaluating specific information from collected data.

[0336] "Special fraud patterns" refer to specific combinations of keywords and phrases that are common to fraudulent activities.

[0337] "Emotion analysis" is the process of automatically determining a user's emotional state from speech data.

[0338] "Configured Contacts" refers to recipients of alert notifications that are pre-registered by the system.

[0339] An "alert notification" is a warning message that sends warning information to designated recipients when potential fraud is detected.

[0340] "Call interruption means" refers to the ability to interrupt and terminate communications during a call.

[0341] An "automated teller machine" is a device used for financial transactions called an ATM.

[0342] A "warning message" is a text or audio notification intended to alert a user or system.

[0343] "Specific conditions" refers to multiple criteria or rules that are set up to make the system alert.

[0344] "Additional restrictions" refers to verification procedures or functional restriction measures implemented in addition to normal operations.

[0345] This invention is a system that collects voice data during a call in real time, detects signs of special fraud, and prevents fraud damage by analyzing the user's emotions. A specific embodiment of this system is described below.

[0346] 1. Collecting and converting audio data into text

[0347] When the device detects the start of a call, it collects real-time audio data during the call through its built-in microphone. The collected audio data is converted into a digital format and sent to a speech recognition engine such as Google Cloud Speech-to-Text. The speech recognition engine converts the audio data into text data. The device then sends the converted text data to a server.

[0348] 2. Text Data Analysis

[0349] The server receives the text data sent from the device and analyzes it using natural language processing (NLP) techniques, such as SpaCy and NLTK. The server analyzes the text data to detect keywords and phrases that indicate fraud, thereby determining whether a fraud pattern exists.

[0350] 3. Emotion analysis

[0351] The device also sends voice data during the call to an emotion analysis engine, such as IBM Watson Tone Analyzer, which analyzes the user's emotions (e.g., alertness or anxiety) from the voice data and sends the results to a server. This emotion analysis result is used to evaluate signs of fraud.

[0352] 4. Enhanced alert notifications

[0353] The server evaluates the likelihood of fraud by combining the results of text analysis and sentiment analysis. If it determines that there is a high possibility of fraud, the server generates an alert notification and sends it to the specified contacts (usually the victim's family or trusted people). This alert notification will inform the user that there is a high possibility of fraud and urge them to take immediate action.

[0354] 5. Call blocking

[0355] If the server determines that there is a high possibility of fraud, the terminal will automatically disconnect the call in response to instructions from the server, thereby preventing the user from continuing the call with the fraudster.

[0356] 6. ATM warning messages

[0357] If there is a high possibility of fraud, the server sends a warning message to the ATM network that the victim may use. The ATM receives the warning message and displays a warning to the user. Furthermore, if certain conditions are met (e.g., large withdrawals), additional restrictions (withdrawal limits and identity verification procedures) are triggered.

[0358] Specific examples

[0359] Example 1: Victim receives a call from a scammer

[0360] 1. Device: The victim's mobile phone receives the call from the scammer. The device collects the audio during the call and converts it into text in real time using Google Cloud Speech-to-Text.

[0361] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[0362] 3. Device: At the same time, the call audio is analyzed for the user's emotions using IBM Watson Tone Analyzer. If alertness or anxiety is detected, that information is also sent to the server.

[0363] 4. Server: Determines the likelihood of fraud and takes into account sentiment analysis results to create an enhanced alert, informing users that a suspected fraudulent call has been detected and recommending immediate action.

[0364] 5. Device: Call blocking is implemented, preventing the victim from continuing the call with the scammer.

[0365] 6. Server: Sends a warning message to ATMs that the victim may use, indicating a high probability of fraud.

[0366] 7. ATMs: ATMs will display warning messages and trigger additional restrictions based on certain conditions.

[0367] Prompt Sentence Examples

[0368] "I want to design a system that analyzes voice data during phone calls in real time, detects signs of fraud, and enhances alert notifications based on sentiment analysis. Please explain the process flow of this system in detail."

[0369] In this way, the present invention is a system that realizes a multi-stage response to prevent fraud damage through real-time collection of voice data, text conversion, fraud detection, emotion analysis, alert notification, call blocking, and ATM warning.

[0370] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0371] Step 1:

[0372] Collecting and converting audio data into text

[0373] When the device detects the start of a call, it collects voice data during the call in real time through its built-in microphone. The input is voice data, which is converted into a digital format and sent to a voice recognition engine such as Google Cloud Speech-to-Text. The data is then processed by converting the voice data into text data using the voice recognition engine, and the output is text data.

[0374] Step 2:

[0375] Text data analysis

[0376] The server receives text data sent from the device. The input is text data, and the server analyzes this data using natural language processing (NLP) techniques such as SpaCy and NLTK. Specifically, it detects keywords and phrases that indicate fraud from the text data. Data processing involves analyzing the content of the text and identifying fraud patterns, and the output is the fraud pattern detection results.

[0377] Step 3:

[0378] Emotion analysis

[0379] The device also sends voice data during a call to an emotion analysis engine such as IBM Watson Tone Analyzer. The input is voice data, which the emotion analysis engine analyzes to determine the user's emotional state. Specifically, emotions such as vigilance and anxiety are extracted from the voice data. Data processing involves emotion analysis, and the emotion analysis results are generated as output. These results are then sent to the server.

[0380] Step 4:

[0381] Enhanced alert notifications

[0382] The server evaluates the likelihood of fraud by combining the results of text analysis and sentiment analysis. The inputs are the results of text analysis and sentiment analysis, and the server uses these to make a comprehensive judgment on the likelihood of fraud. Data calculation involves integrating the results of the two analyses to calculate the probability of fraud occurring, and the output is an alert notification message. The server then sends this alert notification to the configured contacts.

[0383] Step 5:

[0384] Call blocking

[0385] If the server determines that there is a high possibility of fraud, it sends a call-hanging instruction to the terminal. The terminal receives this instruction and automatically ends the current call. The input is the "call-hanging" instruction from the server, and the output is the call termination process. Specifically, the terminal executes the "call-hanging" command to hang up the call, preventing the user from continuing the conversation with the fraudster.

[0386] Step 6:

[0387] ATM warning message

[0388] If there is a high possibility of fraud, the server sends a warning message to the ATM network that the victim may use. The input is the possibility of fraud and related information, and the server generates a warning message based on this. Data processing involves generating a warning message, and as output, a warning message is generated that is sent to the ATM network. The ATM receives this message, displays a warning to the user, and if certain conditions are met (e.g., withdrawing a large amount), additional restrictions are triggered.

[0389] (Application example 2)

[0390] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0391] While conventional call monitoring systems can detect signs of fraud, they have the problem of being unable to analyze the user's emotions and take appropriate action based on the results. Even if signs of fraud are detected, there is a risk that the damage will increase if the user continues the call. Furthermore, even if a fraudulent call is deemed highly likely, the system lacks the functionality to immediately provide appropriate warnings and countermeasures. Therefore, a multi-stage response that takes user emotions into account is needed.

[0392] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data during a call in real time and converting the voice into text format, means for analyzing the converted text data and detecting patterns of special fraud, means for sending an alert notification to a set contact if there is a high possibility of fraud, means for cutting off the call if it is determined that there is a high possibility of fraud, means for sending a warning message to the relevant transaction device, means for analyzing the caller's emotions from the collected voice data, and means for strengthening the content of the alert notification based on the emotion analysis results. This enables multi-stage and rapid response.

[0393] "During a call, audio data" refers to audio information collected through the terminal's microphone while a call is in progress.

[0394] "Text format" refers to a format in which audio data is analyzed and converted into text data.

[0395] "Patterns of special fraud" refers to a collection of text data that includes keywords, phrases, and speaking characteristics specific to fraudulent acts.

[0396] "Configured Contacts" are emergency contacts designated by the user in advance, typically trusted family and friends.

[0397] "Alert Notification" means a warning message sent to a Contact when potential fraud is detected.

[0398] "Call blocking" is an operation that forcibly ends a current call if it is determined that there is a high possibility of fraud.

[0399] "Transaction device" refers to a device that conducts financial transactions, such as an ATM, and includes devices that have the function of receiving warning messages.

[0400] A "warning message" is a notification intended to alert a user or transaction device to suspected fraud.

[0401] "Means for analyzing the caller's emotions from collected voice data" refers to technology that uses voice data to analyze the caller's emotional state (such as vigilance or anxiety).

[0402] "Means to enhance the content of alert notifications based on the results of sentiment analysis" refers to an interface that enhances the importance and urgency of alert notifications based on sentiment analysis, encouraging appropriate action.

[0403] This invention is a system that collects voice data during calls in real time, detects signs of special fraud, and analyzes the user's emotions to prevent fraud damage before it occurs. The specific processing flow and details of the program for this system are explained below.

[0404] 1. Collecting and converting audio data into text

[0405] Device:

[0406] When a call is initiated, the device immediately collects voice data during the call. The voice data is captured through the device's microphone and converted into digital form. This collected voice data is immediately sent to a speech recognition engine (e.g., Google Speech-to-Text API) using highly accurate voice recognition technology to convert the voice into corresponding text data.

[0407] 2. Text Data Analysis

[0408] server:

[0409] The converted text data is sent to a server, which analyzes it using natural language processing (NLP) techniques (e.g., spaCy or NLTK). This analysis process includes algorithms that detect keywords and phrases characteristic of fraud. Based on the analysis results, it is determined whether a fraud pattern exists.

[0410] 3. Emotion analysis

[0411] Device:

[0412] In parallel, the voice data during the call is also sent to an emotion engine (for example, IBM Watson Tone Analyzer). The emotion engine recognizes emotions from the user's voice. If emotions such as alertness or anxiety are detected, that information is also sent to the server.

[0413] 4. Enhanced alert notifications

[0414] server:

[0415] If the analysis determines that there is a high possibility of fraud, the results of the sentiment analysis are also taken into account. If the user's sentiment indicates vigilance or anxiety, the alert notification will be enhanced and a message urging immediate action will be generated. The server will then send the alert notification to the configured contacts (usually the victim's family or trusted individuals). The notification can be sent using the Twilio API.

[0416] 5. Call blocking

[0417] Device:

[0418] If a fraudulent call is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the victim from further interaction with the fraudster.

[0419] 6. ATM warning messages

[0420] server:

[0421] If fraud is likely, the server will send a warning message to transaction devices that the victim may use (e.g., ATM networks), stating that a fraud is suspected and urging them to be careful.

[0422] Trading Device:

[0423] Upon receiving the warning message, the transaction device will display a warning to the user, and if certain conditions are met (e.g., large withdrawals), additional restrictions may be imposed, including withdrawal limits and additional identity verification procedures.

[0424] Specific examples

[0425] Example 1: Victim receives a call from a scammer

[0426] 1. Device: The victim's mobile phone receives a call from a scammer. The device collects the audio during the call and converts it into text in real time.

[0427] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[0428] 3. Device: At the same time, the emotion engine analyzes the user's emotions from the voice data during the call. If alertness or anxiety is detected, that information is also sent to the server.

[0429] 4. Server: Determines the likelihood of fraud and takes into account sentiment analysis results to create an enhanced alert, which includes information that a suspected fraudulent call has been detected and instructions to take immediate action.

[0430] 5. Device: Call blocking is implemented, preventing the victim from continuing the call with the scammer.

[0431] 6. Server: Sends a warning message to transaction devices that the victim may use, indicating that there is a high possibility of fraud.

[0432] 7. Trading Device: The trading device displays warning messages, notifies users of potential fraud, and triggers additional restrictions based on certain conditions.

[0433] Example prompt sentence:

[0434] "This call shows signs of fraud. It contains the following phrases: bank account, large withdrawal, family emergency. The scam was also confirmed through sentiment analysis. End the call immediately and send an alert notification to a trusted contact."

[0435] In this way, the system of the present invention detects fraudulent behavior in real time during a call and takes multi-stage measures by combining it with emotion analysis, thereby making it possible to prevent fraud damage before it occurs.

[0436] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0437] Step 1:

[0438] Device: When a call is initiated, the device immediately collects audio data during the call. Audio data is captured through the device's microphone and converted into a digital format. This collected audio data is converted into text data in real time using the Google Speech-to-Text API.

[0439] Input: Audio data

[0440] Output: Text data

[0441] What it does: The microphone captures voice data, which is then sent to a speech recognition engine, which converts the speech into text and returns it to the device.

[0442] Step 2:

[0443] Terminal: The converted text data is sent to the server.

[0444] Input: Text data

[0445] Output: Text data sent to the server

[0446] Specific operation: The terminal sends the converted text data to the server via the network.

[0447] Step 3:

[0448] Server: The server analyzes the text data using natural language processing (NLP) techniques (e.g., spaCy or NLTK). The analysis process includes algorithms that detect keywords and phrases characteristic of fraud.

[0449] Input: Text data

[0450] Output: High probability of fraud detection results

[0451] What it does: It feeds text data into an NLP engine to check for the presence of fraud-related keywords, and if a fraud pattern is detected, it flags it accordingly.

[0452] Step 4:

[0453] Terminal: In parallel, the voice data during the call is also sent to an emotion engine (e.g., IBM Watson Tone Analyzer), which recognizes emotions from the user's voice.

[0454] Input: Audio data

[0455] Output: Emotion analysis results

[0456] What it does: It sends voice data to an emotion analysis engine, which analyzes the tone and patterns of the voice to identify the emotional state. If it detects emotions such as alertness or anxiety, it sends the results to a server.

[0457] Step 5:

[0458] Server: If the analysis determines that fraud is likely, the sentiment analysis is also taken into account. An enhanced alert notification is generated and sent to configured contacts.

[0459] Input: Fraud pattern detection results and sentiment analysis results

[0460] Output: Enhanced alert notifications

[0461] What it does: Based on the results of sentiment analysis, it generates prompt text to adjust the content and urgency of fraud notifications, and uses the Twilio API to send alert notifications to configured contacts.

[0462] Step 6:

[0463] Device: If a fraudulent call is deemed likely, the current call will be automatically terminated based on instructions from the server.

[0464] Input: Instructions from the server

[0465] Output: End of call

[0466] Specific operation: Upon receiving a call termination command from the server, the device forcibly terminates the call.

[0467] Step 7:

[0468] Server: If fraud is suspected, it sends a warning message to the victim's potential transaction device. This message notifies the victim of the suspected fraud.

[0469] Input: Fraud pattern detection results

[0470] Output: Warning message to trading device

[0471] What it does: If fraud is suspected, a warning message is sent via a custom API to a transaction device (e.g., an ATM), which displays the message and, if necessary, initiates additional restrictive measures.

[0472] In this way, the system of the present invention performs the necessary data processing and calculation at each step, making it possible to prevent fraud damage through multi-stage responses.

[0473] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0474] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0475] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0476] [Second embodiment]

[0477] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0478] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0479] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0480] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0481] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0482] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0483] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0484] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0485] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0486] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0487] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0488] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0489] The present invention is a system that collects voice data during a call in real time, detects signs of special fraud, and prevents fraud damage before it occurs. The specific processing flow and details of the program for this system are explained below.

[0490] 1. Collecting and converting audio data into text

[0491] Device:

[0492] When a call is initiated, the device collects real-time audio data during the call. Specifically, it uses the device's microphone to record the audio and stores it digitally. The collected audio data is immediately sent to a speech recognition engine, which converts the audio into corresponding text data. This conversion process is carried out using highly accurate speech recognition technology.

[0493] 2. Text Data Analysis

[0494] server:

[0495] The converted text data is sent to a server. The server receives the text data and analyzes it using natural language processing (NLP) techniques. This analysis includes algorithms that detect keywords and phrases characteristic of fraud. Based on the analysis results, it determines whether a fraud pattern exists.

[0496] 3. Sending alert notifications

[0497] server:

[0498] If the analysis determines that a call is likely to be fraudulent, the server immediately sends an alert notification to configured contacts (usually the victim's family or trusted individuals) warning them that a suspected fraudulent call has been detected and recommending actions to take to prevent it.

[0499] 4. Call blocking

[0500] Device:

[0501] If a fraudulent call is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the victim from further interaction with the fraudster.

[0502] 5. ATM warning messages

[0503] server:

[0504] If fraud is likely, the server will send a warning message to ATM networks that the victim may use, stating that a fraud attempt is suspected and urging caution.

[0505] ATM:

[0506] When an ATM receives a warning message, it will display a warning to the user and, if certain conditions are met (e.g., large withdrawals), will trigger additional restrictions, including withdrawal limits and additional identity verification procedures.

[0507] Specific examples

[0508] Example 1: Victim receives a call from a scammer

[0509] 1. Device: The victim's mobile phone receives a call from a scammer. The device collects the audio during the call and converts it into text in real time.

[0510] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[0511] 3. Server: Determines that the call is likely fraudulent and sends an alert to the victim's family, informing them that a suspected fraudulent call has been detected and instructing them to take immediate action.

[0512] 4. Device: Call blocking is implemented, preventing the victim from continuing the call with the scammer.

[0513] 5. Server: Sends a warning message to ATMs that the victim may use, indicating that there is a high possibility of fraud.

[0514] 6. ATMs: ATMs display warning messages to alert users to potential fraud and trigger additional restrictions based on certain conditions.

[0515] In this way, the system of the present invention can detect fraudulent activity in real time during a call and take multi-stage measures to prevent fraud damage before it occurs.

[0516] The processing flow will be explained below.

[0517] Step 1:

[0518] Device:

[0519] When a call is initiated, the device immediately collects audio data during the call, which is captured through the device's microphone and converted into a digital format.

[0520] Step 2:

[0521] Device:

[0522] The collected voice data is converted into text format in real time. A voice recognition engine is activated, processing the voice data at high speed to generate text data.

[0523] Step 3:

[0524] Device:

[0525] The converted text data is sent to the server, where it is encrypted and transferred using a secure communication protocol.

[0526] Step 4:

[0527] server:

[0528] The server receives the text data and analyzes it using a natural language processing (NLP) engine, which includes detecting keywords and phrases that may indicate fraud.

[0529] Step 5:

[0530] server:

[0531] Based on the analysis results, it is determined whether there is a possibility of fraud. If the probability of fraud exceeds a certain threshold, it is determined that there is a high possibility of fraud.

[0532] Step 6:

[0533] server:

[0534] If a fraudulent call is detected, an alert notification will be sent to your configured contacts, containing a message informing you that a suspected fraudulent call has been detected.

[0535] Step 7:

[0536] server:

[0537] After sending the alert notification, the server sends a command to the device to disconnect the call, which is also encrypted and communicated securely.

[0538] Step 8:

[0539] Device:

[0540] The device receives instructions from the server and automatically ends the current call, preventing the victim from continuing the call with the scammer.

[0541] Step 9:

[0542] server:

[0543] Assuming there is a high probability of fraud, the server will send a warning message to the affected ATM, which will prevent the victim from immediately withdrawing cash.

[0544] Step 10:

[0545] ATM:

[0546] When an ATM receives a warning message, it will display a warning accordingly, and if the user meets certain conditions (for example, if they attempt to withdraw a large amount), additional restrictions will be automatically implemented.

[0547] Example 1

[0548] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0549] Currently, the number of victims of special frauds is on the rise, and the methods targeting the elderly in particular are becoming more sophisticated. An effective system to prevent such fraud is needed, but existing systems have difficulty responding in real time, and countermeasures are often only implemented after a crime has occurred. For this reason, there is an urgent need to develop a system that can detect signs of fraud in real time during a call and respond immediately.

[0550] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0551] In this invention, the server includes means for collecting voice data during a call in real time and converting the voice into text format, means for analyzing the converted text data using natural language processing technology to detect patterns of special fraud, and means for creating prompt sentences using a generative model to evaluate the possibility of fraud. This makes it possible to detect signs of special fraud in real time during a call and immediately send an alert or block the call.

[0552] "During a call, voice data" refers to digital data of voice collected using a microphone while a user is making a call.

[0553] "Means for converting to text" refers to the speech recognition technology or software used to convert collected voice data into text.

[0554] "Natural language processing technology" is an artificial intelligence technology that analyzes text data, extracts information through an understanding of vocabulary, grammar, and meaning, and recognizes specific patterns.

[0555] "Special fraud patterns" are a collection of typical phrases, keywords, and behavioral patterns used when committing fraud.

[0556] "Means for sending alert notifications" means communications methods or technologies for sending real-time warning messages to designated contacts in the event of a high likelihood of fraud.

[0557] "Call blocking measures" are technologies or devices that forcibly terminate an ongoing call when it is determined that there is a high possibility of fraud.

[0558] "Means for sending warning messages" refers to technology for sending warning messages to automated teller machine networks that victims may use when signs of fraud are detected.

[0559] "Means for activating additional restrictions" means a function or device for imposing withdrawal limits on transactions or implementing additional identity verification procedures in accordance with certain conditions.

[0560] "Means for creating prompt sentences to assess the likelihood of fraud using a generative model" refers to a technology or method that uses an artificial intelligence model to assess the likelihood of fraud based on the frequency of occurrence of specific phrases or keywords in text data, and generates prompt sentences to present the results to users or administrators.

[0561] The present invention is a system for collecting voice data during a call in real time, detecting signs of special fraud, and preventing fraud damage before it occurs. The system of the present invention is mainly composed of terminals, a server, and an ATM.

[0562] Collecting and converting audio data into text

[0563] Device:

[0564] When a call is initiated, the device collects real-time audio data during the call. Specifically, it uses the device's microphone to record the audio and stores it digitally. The collected audio data is immediately sent to a speech recognition engine, which converts the audio into corresponding text data. This conversion process is carried out using high-precision speech recognition technology, such as Google Cloud Speech-to-Text.

[0565] Text data analysis

[0566] server:

[0567] The converted text data is sent to a server. The server receives the text data and analyzes it using natural language processing (NLP) techniques. This analysis includes generative AI models (e.g., BERT or GPT-3) to detect keywords and phrases characteristic of fraud. Based on the analysis results, it determines whether a fraud pattern exists.

[0568] Sending alert notifications

[0569] server:

[0570] If the analysis determines that a call is likely to be fraudulent, the server immediately sends an alert notification to configured contacts (usually the victim's family or trusted individuals) warning them that a suspected fraudulent call has been detected and recommending actions to take to prevent it.

[0571] Call blocking

[0572] Device:

[0573] If fraud is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the user from further interaction with the fraudster.

[0574] ATM warning message

[0575] server:

[0576] If fraud is likely, the server will send a warning message to ATM networks that the victim may use, stating that a fraud attempt is suspected and urging caution.

[0577] ATM:

[0578] When an ATM receives a warning message, it will display a warning to the user and, if certain conditions are met (e.g., large withdrawals), will trigger additional restrictions, including withdrawal limits and additional identity verification procedures.

[0579] Specific examples

[0580] Example 1: Victim receives a call from a scammer

[0581] 1. Device: Imagine a scenario where a victim's mobile phone receives a call from a scammer. The device collects the audio during the call and converts it into text in real time.

[0582] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[0583] 3. Server: Determines that the call is likely fraudulent and sends an alert notification to the victim's family, informing them that a suspected fraudulent call has been detected and providing instructions for immediate action.

[0584] 4. Device: Call blocking is performed, preventing the victim from continuing the conversation with the scammer.

[0585] 5. Server: Determines that there is a high possibility of fraud and sends a warning message to ATMs that the victim may use.

[0586] 6. ATMs: ATMs will display warning messages to notify users of potential fraud, and additional restrictions will be implemented if certain conditions are met.

[0587] Prompt Sentence Examples

[0588] Prompt: "Decide if this call is a scam"

[0589] Generative AI models analyze the frequency of specific keywords and phrases to assess the likelihood of fraud and, based on the results, recommend countermeasures in cases where fraud is suspected.

[0590] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0591] Step 1:

[0592] Start of voice data collection

[0593] Device:

[0594] When a user starts a call, the device detects the start of the call. At this time, the device's microphone becomes active and collects audio data during the call in real time. The collected audio data is saved in digital format. The input is the user's voice during the call, and the output is the collected digital audio data. Specifically, when the user presses the answer button, the device detects the start of the call using an event listener.

[0595] Step 2:

[0596] Converting audio data to text

[0597] Device:

[0598] The device sends the collected voice data to a voice recognition engine, which (for example, Google Cloud Speech-to-Text API) converts the voice into text data in real time. The input is digital voice data, and the output is the corresponding text data. Specifically, the voice recorded by the microphone is stored in digital format and sent to the voice recognition engine.

[0599] Step 3:

[0600] Sending text data

[0601] Device:

[0602] The converted text data is sent to the server. The input is text data, and the output is data transmission to the server. Specifically, the terminal sends the text data to the server using an HTTP request.

[0603] Step 4:

[0604] Text data analysis

[0605] server:

[0606] The server analyzes the received text data using natural language processing technology. This analysis uses a generative AI model that detects keywords and phrases characteristic of fraud. The input is text data, and the output is the analysis result (whether or not there is a possibility of fraud). Specifically, the server analyzes the text data using a generative AI model such as BERT or GPT-3.

[0607] Step 5:

[0608] Assessment of the likelihood of fraud

[0609] server:

[0610] The server uses a generative AI model to assess the likelihood of fraud and create a prompt. The input is analyzed text data, and the output is a prompt that assesses the likelihood of fraud. Specifically, the generative AI model evaluates specific keywords and phrases and generates a prompt based on the results.

[0611] Step 6:

[0612] Sending alert notifications

[0613] server:

[0614] If the server determines that there is a high possibility of fraud, it will send an alert notification to the configured contacts. The input is the prompt text and the alert notification destination information, and the output is sending the alert notification. Specifically, the server will send the notification using SMS or email API.

[0615] Step 7:

[0616] Call blocking

[0617] Device:

[0618] Based on instructions from the server, the terminal automatically disconnects the current call. The input is a call disconnect command from the server, and the output is the termination of the call. Specifically, the terminal uses the telephone communication module to forcibly terminate the call.

[0619] Step 8:

[0620] Sending warning messages to ATMs

[0621] server:

[0622] If the server determines that there is a high possibility of fraud, it sends a warning message to the ATM network that the victim may use. The input is a warning message and ATM network information, and the output is the transmission of the warning message. Specifically, the server sends a message to the ATM network using a network protocol.

[0623] Step 9:

[0624] ATM warning signs and restrictions

[0625] ATM:

[0626] When an ATM receives a warning message, it displays the warning to the user. Furthermore, if certain conditions are met (e.g., withdrawing a large amount), additional restrictions are implemented. The input is the received warning message and transaction conditions, and the output is the display of the warning and the implementation of restrictions. Specifically, the ATM displays the warning message on its display and, if necessary, performs additional identity verification procedures.

[0627] (Application example 1)

[0628] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0629] In recent years, there has been an increase in specialized frauds using sophisticated methods, and many people have fallen victim to them. In particular, frauds committed over the phone often target the elderly and those with little financial knowledge, so fast and effective countermeasures are needed. However, current systems lack real-time fraud detection capabilities, making it difficult to prevent damage before it occurs. For this reason, there is a need for the development of a system that can quickly detect possible fraud during a call and take appropriate countermeasures.

[0630] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0631] In this invention, the server includes means for collecting voice data during a call in real time and converting the voice into text format, means for analyzing the converted text data and detecting patterns of special fraud, means for sending an alert notification to a set contact if there is a high possibility of fraud, means for blocking the call if it is determined that there is a high possibility of fraud, means for sending a warning message to the relevant ATM, means for displaying the warning message on the user's mobile device, and means for analyzing the text data using a generative AI model. This makes it possible to detect special frauds occurring during a call in real time and take prompt measures.

[0632] "Voice data" refers to digital data of voice collected during a call using a microphone or the like of a terminal.

[0633] "Text format" is a digital data format in which voice data is converted into a string of characters.

[0634] "Converted text data" refers to data obtained by converting voice data into text format using voice recognition technology.

[0635] "Patterns of special fraud" are characteristic patterns in text data that contain keywords, phrases, and structures specific to fraudulent activity.

[0636] An "alert notification" is a warning message sent to configured contacts when potential fraud is detected.

[0637] "Call blocking measures" are features that automatically terminate an active call if there is a high likelihood of fraud.

[0638] A "warning message" is a message that warns of possible fraud.

[0639] An "ATM (Automated Teller Machine)" is an automated machine for conducting financial transactions.

[0640] "Means for displaying a warning message" refers to a function that visually displays a warning message on an ATM or a user's mobile device.

[0641] A "generative AI model" is an artificial intelligence model trained to analyze text data using natural language processing and detect fraudulent patterns.

[0642] The present invention is a system for collecting voice data during a call in real time, detecting signs of special fraud, and preventing fraud damage before it occurs. Detailed embodiments of this system are described below.

[0643] Collecting and converting audio data into text

[0644] Device:

[0645] When a call is initiated, the device uses the smartphone's microphone to collect real-time audio data during the call and stores it digitally. This audio data is then immediately sent to a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the audio into text data.

[0646] Text data analysis

[0647] server:

[0648] The converted text data is then sent to a cloud server, which receives the data and analyzes it using natural language processing (NLP) techniques, using a generative AI model (e.g., OpenAI's GPT-4) to detect keywords and phrases characteristic of fraud.

[0649] Sending alert notifications

[0650] server:

[0651] If the analysis determines that there is a high possibility of fraud, the server immediately sends an alert notification to the user's configured contacts (usually family members or trusted individuals) that warns of the suspected fraud and provides advice on how to respond.

[0652] Call blocking

[0653] Device:

[0654] If a fraudulent call is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the user from further interaction with the fraudster.

[0655] Local Warning Display

[0656] Device:

[0657] After the call is blocked, the user will receive a warning message on their mobile device, including a message about suspected fraud and next steps to take (e.g., contacting police or suspending their card).

[0658] Sending warning messages to ATMs

[0659] server:

[0660] If fraud is likely, the server will send a warning message to ATM networks that the victim may use, stating that a fraud attempt is suspected and urging caution.

[0661] ATM:

[0662] When an ATM receives a warning message, it will display a warning to the user and, if certain conditions are met (e.g., large withdrawals), will trigger additional restrictions, including withdrawal limits and additional identity verification procedures.

[0663] Specific examples

[0664] User Scenarios

[0665] 1. Call Initiation: The user receives a call from an unknown number.

[0666] 2. Speech recognition: During a call, the smartphone app converts voice into text data.

[0667] 3. NLP analysis: The text data is sent to a cloud server and analyzed using the GPT-4 model.

[0668] 4. Alerts: Keyword such as "bank account" or "transfer" are detected and an alert is sent to family members. If necessary, the call will be blocked.

[0669] 5. Local Alert: After the call ends, the user will be shown a warning message and guided on next steps.

[0670] Prompt Sentence Examples

[0671] "The text data of a suspected fraudulent call is shown below. Analyze this text data and determine whether it is likely to be fraudulent. If you are suspicious, output a warning message. Text data: 'A family member is in the hospital in an emergency. I need money immediately, so please transfer it to my bank account.'"

[0672] In this way, the present invention is a system that can detect fraudulent acts during calls in real time and prevent damage by taking multi-stage measures.

[0673] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0674] Step 1:

[0675] Collecting and converting audio data into text

[0676] Device:

[0677] When a call is initiated, the device uses a microphone to collect voice data in real time, stores the voice data in digital format, and immediately sends it to a voice recognition engine (e.g., Google Cloud Speech-to-Text API). Voice data is input and corresponding text data is output. At this time, the voice data is temporarily stored in the device's storage.

[0678] Step 2:

[0679] Sending and analyzing text data

[0680] server:

[0681] The server receives the text data sent from the device. The server sends this text data to a natural language processing (NLP) engine (e.g., OpenAI's GPT-4) to detect characteristic fraud keywords and phrases. The server receives the text data as input and outputs an analysis result indicating the likelihood of fraud.

[0682] Step 3:

[0683] Sending alert notifications

[0684] server:

[0685] If the analysis result indicates a high probability of fraud, the server will send an alert notification to the configured contacts. The alert notification will include information about the suspected fraud and the user's recommended actions. The server takes the analysis result as input and sends an alert notification to the contacts as output.

[0686] Step 4:

[0687] Call blocking

[0688] Device:

[0689] When the device receives a call-hangup command from the server, it automatically hangs up the current call, preventing the user from continuing to interact with the scammer. It takes a command from the server as input and ends the call as output.

[0690] Step 5:

[0691] Local Warning Display

[0692] Device:

[0693] After the call is blocked, the device displays a warning message on the user's screen, informing them of the suspected fraud and providing next steps. The input is the completion of the call blocking, and the output is to display a warning message on the device.

[0694] Step 6:

[0695] Sending warning messages to ATMs

[0696] server:

[0697] If the server determines that there is a particularly high possibility of fraud, it sends a warning message to the ATM network, stating that the user should exercise caution. The server takes as input data indicating a high possibility of fraud, and outputs by sending a warning message to the ATM network.

[0698] Step 7:

[0699] ATM warning message display and restriction measures

[0700] ATM:

[0701] When an ATM receives a warning message, it displays a visual warning to the user. If certain conditions are met, such as a large withdrawal, additional withdrawal restrictions or identity verification procedures are implemented. The input is a warning message from the server, and the output is the display of a warning message or the implementation of a restriction.

[0702] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0703] This invention is a system that collects voice data during calls in real time, detects signs of special fraud, and analyzes the user's emotions to prevent fraud damage before it occurs. The specific processing flow and details of the program for this system are explained below.

[0704] 1. Collecting and converting audio data into text

[0705] Device:

[0706] When a call is initiated, the device immediately collects voice data during the call. The voice data is picked up through the device's microphone and converted into digital form. The collected voice data is immediately sent to a speech recognition engine, which converts the voice into corresponding text data. This conversion process is carried out using highly accurate voice recognition technology.

[0707] 2. Text Data Analysis

[0708] server:

[0709] The converted text data is sent to a server. The server receives the text data and analyzes it using natural language processing (NLP) techniques. This analysis includes algorithms that detect keywords and phrases characteristic of fraud. Based on the analysis results, it determines whether a fraud pattern exists.

[0710] 3. Emotion analysis

[0711] Device:

[0712] Voice data during a call is also sent to the emotion engine, which recognizes emotions from the user's voice. If emotions such as alertness or anxiety are detected, that information is also sent to the server.

[0713] 4. Enhanced alert notifications

[0714] server:

[0715] If the analysis determines that a scam is likely, the results of the sentiment analysis are also taken into account. In particular, if the user's sentiment indicates vigilance or anxiety, the alert notification will be enhanced and a message urging immediate action will be generated. The server will then send the alert notification to the designated contacts (usually the victim's family or trusted individuals).

[0716] 5. Call blocking

[0717] Device:

[0718] If a fraudulent call is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the victim from further interaction with the fraudster.

[0719] 6. ATM warning messages

[0720] server:

[0721] If fraud is likely, the server will send a warning message to ATM networks that the victim may use, stating that a fraud attempt is suspected and urging caution.

[0722] ATM:

[0723] When an ATM receives a warning message, it will display a warning to the user and, if certain conditions are met (e.g., large withdrawals), will trigger additional restrictions, including withdrawal limits and additional identity verification procedures.

[0724] Specific examples

[0725] Example 1: Victim receives a call from a scammer

[0726] 1. Device: The victim's mobile phone receives a call from a scammer. The device collects the audio during the call and converts it into text in real time.

[0727] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[0728] 3. Device: At the same time, the emotion engine analyzes the user's emotions from the voice data during the call. If alertness or anxiety is detected, that information is also sent to the server.

[0729] 4. Server: Determines the likelihood of fraud and takes into account sentiment analysis results to create an enhanced alert, which includes information that a suspected fraudulent call has been detected and instructions to take immediate action.

[0730] 5. Device: Call blocking is implemented, preventing the victim from continuing the call with the scammer.

[0731] 6. Server: Sends a warning message to ATMs that the victim may use, indicating a high probability of fraud.

[0732] 7. ATMs: ATMs display warning messages to alert users to potential fraud and trigger additional restrictions based on certain conditions.

[0733] In this way, the system of the present invention detects fraudulent behavior in real time during a call and takes multi-stage measures by combining it with emotion analysis, thereby making it possible to prevent fraud damage before it occurs.

[0734] The processing flow will be explained below.

[0735] Step 1:

[0736] Device:

[0737] When a call is initiated, the device immediately begins collecting audio data during the call. It uses the device's microphone to capture audio and stores it digitally. It is set to collect audio data continuously.

[0738] Step 2:

[0739] Device:

[0740] The collected voice data is sent to a voice recognition engine in real time and instantly converted into text. The voice recognition engine uses a highly accurate recognition algorithm to generate text data from the voice. The generated text data is then stored in memory.

[0741] Step 3:

[0742] Device:

[0743] The converted text data is encrypted and sent to a server using a secure communication protocol, where it is transferred in real time to the server for analysis.

[0744] Step 4:

[0745] server:

[0746] The server receives the text data, which is then analyzed using a natural language processing (NLP) engine to detect keywords and phrases that indicate fraud.

[0747] Step 5:

[0748] server:

[0749] Based on the analysis results, we assess whether there is a possibility of fraud, using pre-defined criteria and pattern matching algorithms to determine whether there is a high probability of fraud.

[0750] Step 6:

[0751] Device:

[0752] Voice data during a call is simultaneously sent to the emotion engine, which analyzes the user's voice tone, tempo, and other factors to recognize their emotions. Emotion analysis results are generated in real time.

[0753] Step 7:

[0754] Device:

[0755] If the emotion analysis results indicate alarm or anxiety, the results are also encrypted and sent to the server. The emotion information is used as part of the fraud detection algorithm.

[0756] Step 8:

[0757] server:

[0758] If a fraudulent call is deemed likely, the results of the sentiment analysis are taken into account to enhance the alert notification, which includes a message informing users that a suspected fraudulent call has been detected and recommending immediate action.

[0759] Step 9:

[0760] server:

[0761] An alert notification is sent to configured contacts, typically family members or trusted associates of the victim, and the notification is sent immediately.

[0762] Step 10:

[0763] Device:

[0764] It receives call blocking instructions from the server and automatically ends the current call, preventing the victim from having any further contact with the scammer.

[0765] Step 11:

[0766] server:

[0767] If there is a high possibility of fraud, the server sends a warning message to the ATM, urging caution due to suspected fraud.

[0768] Step 12:

[0769] ATM:

[0770] When an ATM receives a warning message, it will display a warning accordingly. If the user meets certain conditions (e.g., withdrawing a large amount), additional restrictions will be triggered. In addition, withdrawal restrictions and additional identity verification procedures will be implemented.

[0771] In this way, the system of the present invention can detect fraudulent activity in real time during a call and take multi-stage measures to prevent fraud damage before it occurs. Furthermore, by analyzing the user's emotions, it can provide more accurate alerts and promote a quicker response.

[0772] Example 2

[0773] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0774] In today's society, where fraud damage is on the rise, there is a growing need for a system that can detect special frauds that occur during phone calls in real time and combine them with user emotion analysis to quickly and accurately prevent fraud. Conventional methods have made it difficult to detect signs of fraud in real time, making it difficult to prevent damage before it occurs. In addition, there is a demand for more accurate fraud detection by incorporating user emotion analysis.

[0775] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0776] In this invention, the server includes means for collecting voice data during a call in real time and converting the voice into text format, means for analyzing the converted text data and detecting patterns of special fraud, and means for analyzing emotions from the collected voice data in real time. This makes it possible to detect signs of special fraud occurring during a call in real time and to respond quickly and accurately based on the analysis of the user's emotions.

[0777] "Voice data during a call" refers to digital data of the voice collected through the microphone of the terminal while a call is being made.

[0778] "Real-time collection" refers to the process of capturing audio data on the fly without delay.

[0779] The "means for converting voice into text format" is a mechanism for converting voice data into corresponding character string data using voice recognition technology.

[0780] "Converted text data" is character string data generated by speech recognition technology.

[0781] "Analysis" is the process of extracting and evaluating specific information from collected data.

[0782] "Special fraud patterns" refer to specific combinations of keywords and phrases that are common to fraudulent activities.

[0783] "Emotion analysis" is the process of automatically determining a user's emotional state from speech data.

[0784] "Configured Contacts" refers to recipients of alert notifications that are pre-registered by the system.

[0785] An "alert notification" is a warning message that sends warning information to designated recipients when potential fraud is detected.

[0786] "Call interruption means" refers to the ability to interrupt and terminate communications during a call.

[0787] An "automated teller machine" is a device used for financial transactions called an ATM.

[0788] A "warning message" is a text or audio notification intended to alert a user or system.

[0789] "Specific conditions" refers to multiple criteria or rules that are set up to make the system alert.

[0790] "Additional restrictions" refers to verification procedures or functional restriction measures implemented in addition to normal operations.

[0791] This invention is a system that collects voice data during a call in real time, detects signs of special fraud, and prevents fraud damage by analyzing the user's emotions. A specific embodiment of this system is described below.

[0792] 1. Collecting and converting audio data into text

[0793] When the device detects the start of a call, it collects real-time audio data during the call through its built-in microphone. The collected audio data is converted into a digital format and sent to a speech recognition engine such as Google Cloud Speech-to-Text. The speech recognition engine converts the audio data into text data. The device then sends the converted text data to a server.

[0794] 2. Text Data Analysis

[0795] The server receives the text data sent from the device and analyzes it using natural language processing (NLP) techniques, such as SpaCy and NLTK. The server analyzes the text data to detect keywords and phrases that indicate fraud, thereby determining whether a fraud pattern exists.

[0796] 3. Emotion analysis

[0797] The device also sends voice data during the call to an emotion analysis engine, such as IBM Watson Tone Analyzer, which analyzes the user's emotions (e.g., alertness or anxiety) from the voice data and sends the results to a server. This emotion analysis result is used to evaluate signs of fraud.

[0798] 4. Enhanced alert notifications

[0799] The server evaluates the likelihood of fraud by combining the results of text analysis and sentiment analysis. If it determines that there is a high possibility of fraud, the server generates an alert notification and sends it to the specified contacts (usually the victim's family or trusted people). This alert notification will inform the user that there is a high possibility of fraud and urge them to take immediate action.

[0800] 5. Call blocking

[0801] If the server determines that there is a high possibility of fraud, the terminal will automatically disconnect the call in response to instructions from the server, thereby preventing the user from continuing the call with the fraudster.

[0802] 6. ATM warning messages

[0803] If there is a high possibility of fraud, the server sends a warning message to the ATM network that the victim may use. The ATM receives the warning message and displays a warning to the user. Furthermore, if certain conditions are met (e.g., large withdrawals), additional restrictions (withdrawal limits and identity verification procedures) are triggered.

[0804] Specific examples

[0805] Example 1: Victim receives a call from a scammer

[0806] 1. Device: The victim's mobile phone receives the call from the scammer. The device collects the audio during the call and converts it into text in real time using Google Cloud Speech-to-Text.

[0807] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[0808] 3. Device: At the same time, the call audio is analyzed for the user's emotions using IBM Watson Tone Analyzer. If alertness or anxiety is detected, that information is also sent to the server.

[0809] 4. Server: Determines the likelihood of fraud and takes into account sentiment analysis results to create an enhanced alert, informing users that a suspected fraudulent call has been detected and recommending immediate action.

[0810] 5. Device: Call blocking is implemented, preventing the victim from continuing the call with the scammer.

[0811] 6. Server: Sends a warning message to ATMs that the victim may use, indicating a high probability of fraud.

[0812] 7. ATMs: ATMs will display warning messages and trigger additional restrictions based on certain conditions.

[0813] Prompt Sentence Examples

[0814] "I want to design a system that analyzes voice data during phone calls in real time, detects signs of fraud, and enhances alert notifications based on sentiment analysis. Please explain the process flow of this system in detail."

[0815] In this way, the present invention is a system that realizes a multi-stage response to prevent fraud damage through real-time collection of voice data, text conversion, fraud detection, emotion analysis, alert notification, call blocking, and ATM warning.

[0816] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0817] Step 1:

[0818] Collecting and converting audio data into text

[0819] When the device detects the start of a call, it collects voice data during the call in real time through its built-in microphone. The input is voice data, which is converted into a digital format and sent to a voice recognition engine such as Google Cloud Speech-to-Text. The data is then processed by converting the voice data into text data using the voice recognition engine, and the output is text data.

[0820] Step 2:

[0821] Text data analysis

[0822] The server receives text data sent from the device. The input is text data, and the server analyzes this data using natural language processing (NLP) techniques such as SpaCy and NLTK. Specifically, it detects keywords and phrases that indicate fraud from the text data. Data processing involves analyzing the content of the text and identifying fraud patterns, and the output is the fraud pattern detection results.

[0823] Step 3:

[0824] Emotion analysis

[0825] The device also sends voice data during a call to an emotion analysis engine such as IBM Watson Tone Analyzer. The input is voice data, which the emotion analysis engine analyzes to determine the user's emotional state. Specifically, emotions such as vigilance and anxiety are extracted from the voice data. Data processing involves emotion analysis, and the emotion analysis results are generated as output. These results are then sent to the server.

[0826] Step 4:

[0827] Enhanced alert notifications

[0828] The server evaluates the likelihood of fraud by combining the results of text analysis and sentiment analysis. The inputs are the results of text analysis and sentiment analysis, and the server uses these to make a comprehensive judgment on the likelihood of fraud. Data calculation involves integrating the results of the two analyses to calculate the probability of fraud occurring, and the output is an alert notification message. The server then sends this alert notification to the configured contacts.

[0829] Step 5:

[0830] Call blocking

[0831] If the server determines that there is a high possibility of fraud, it sends a call-hanging instruction to the terminal. The terminal receives this instruction and automatically ends the current call. The input is the "call-hanging" instruction from the server, and the output is the call termination process. Specifically, the terminal executes the "call-hanging" command to hang up the call, preventing the user from continuing the conversation with the fraudster.

[0832] Step 6:

[0833] ATM warning message

[0834] If there is a high possibility of fraud, the server sends a warning message to the ATM network that the victim may use. The input is the possibility of fraud and related information, and the server generates a warning message based on this. Data processing involves generating a warning message, and as output, a warning message is generated that is sent to the ATM network. The ATM receives this message, displays a warning to the user, and if certain conditions are met (e.g., withdrawing a large amount), additional restrictions are triggered.

[0835] (Application example 2)

[0836] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0837] While conventional call monitoring systems can detect signs of fraud, they have the problem of being unable to analyze the user's emotions and take appropriate action based on the results. Even if signs of fraud are detected, there is a risk that the damage will increase if the user continues the call. Furthermore, even if a fraudulent call is deemed highly likely, the system lacks the functionality to immediately provide appropriate warnings and countermeasures. Therefore, a multi-stage response that takes user emotions into account is needed.

[0838] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data during a call in real time and converting the voice into text format, means for analyzing the converted text data and detecting patterns of special fraud, means for sending an alert notification to a set contact if there is a high possibility of fraud, means for cutting off the call if it is determined that there is a high possibility of fraud, means for sending a warning message to the relevant transaction device, means for analyzing the caller's emotions from the collected voice data, and means for strengthening the content of the alert notification based on the emotion analysis results. This enables multi-stage and rapid response.

[0839] "During a call, audio data" refers to audio information collected through the terminal's microphone while a call is in progress.

[0840] "Text format" refers to a format in which audio data is analyzed and converted into text data.

[0841] "Patterns of special fraud" refers to a collection of text data that includes keywords, phrases, and speaking characteristics specific to fraudulent acts.

[0842] "Configured Contacts" are emergency contacts designated by the user in advance, typically trusted family and friends.

[0843] "Alert Notification" means a warning message sent to a Contact when potential fraud is detected.

[0844] "Call blocking" is an operation that forcibly ends a current call if it is determined that there is a high possibility of fraud.

[0845] "Transaction device" refers to a device that conducts financial transactions, such as an ATM, and includes devices that have the function of receiving warning messages.

[0846] A "warning message" is a notification intended to alert a user or transaction device to suspected fraud.

[0847] "Means for analyzing the caller's emotions from collected voice data" refers to technology that uses voice data to analyze the caller's emotional state (such as vigilance or anxiety).

[0848] "Means to enhance the content of alert notifications based on the results of sentiment analysis" refers to an interface that enhances the importance and urgency of alert notifications based on sentiment analysis, encouraging appropriate action.

[0849] This invention is a system that collects voice data during calls in real time, detects signs of special fraud, and analyzes the user's emotions to prevent fraud damage before it occurs. The specific processing flow and details of the program for this system are explained below.

[0850] 1. Collecting and converting audio data into text

[0851] Device:

[0852] When a call is initiated, the device immediately collects voice data during the call. The voice data is captured through the device's microphone and converted into digital form. This collected voice data is immediately sent to a speech recognition engine (e.g., Google Speech-to-Text API) using highly accurate voice recognition technology to convert the voice into corresponding text data.

[0853] 2. Text Data Analysis

[0854] server:

[0855] The converted text data is sent to a server, which analyzes it using natural language processing (NLP) techniques (e.g., spaCy or NLTK). This analysis process includes algorithms that detect keywords and phrases characteristic of fraud. Based on the analysis results, it is determined whether a fraud pattern exists.

[0856] 3. Emotion analysis

[0857] Device:

[0858] In parallel, the voice data during the call is also sent to an emotion engine (for example, IBM Watson Tone Analyzer). The emotion engine recognizes emotions from the user's voice. If emotions such as alertness or anxiety are detected, that information is also sent to the server.

[0859] 4. Enhanced alert notifications

[0860] server:

[0861] If the analysis determines that there is a high possibility of fraud, the results of the sentiment analysis are also taken into account. If the user's sentiment indicates vigilance or anxiety, the alert notification will be enhanced and a message urging immediate action will be generated. The server will then send the alert notification to the configured contacts (usually the victim's family or trusted individuals). The notification can be sent using the Twilio API.

[0862] 5. Call blocking

[0863] Device:

[0864] If a fraudulent call is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the victim from further interaction with the fraudster.

[0865] 6. ATM warning messages

[0866] server:

[0867] If fraud is likely, the server will send a warning message to transaction devices that the victim may use (e.g., ATM networks), stating that a fraud is suspected and urging them to be careful.

[0868] Trading Device:

[0869] Upon receiving the warning message, the transaction device will display a warning to the user, and if certain conditions are met (e.g., large withdrawals), additional restrictions may be imposed, including withdrawal limits and additional identity verification procedures.

[0870] Specific examples

[0871] Example 1: Victim receives a call from a scammer

[0872] 1. Device: The victim's mobile phone receives a call from a scammer. The device collects the audio during the call and converts it into text in real time.

[0873] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[0874] 3. Device: At the same time, the emotion engine analyzes the user's emotions from the voice data during the call. If alertness or anxiety is detected, that information is also sent to the server.

[0875] 4. Server: Determines the likelihood of fraud and takes into account sentiment analysis results to create an enhanced alert, which includes information that a suspected fraudulent call has been detected and instructions to take immediate action.

[0876] 5. Device: Call blocking is implemented, preventing the victim from continuing the call with the scammer.

[0877] 6. Server: Sends a warning message to transaction devices that the victim may use, indicating that there is a high possibility of fraud.

[0878] 7. Trading Device: The trading device displays warning messages, notifies users of potential fraud, and triggers additional restrictions based on certain conditions.

[0879] Example prompt sentence:

[0880] "This call shows signs of fraud. It contains the following phrases: bank account, large withdrawal, family emergency. The scam was also confirmed through sentiment analysis. End the call immediately and send an alert notification to a trusted contact."

[0881] In this way, the system of the present invention detects fraudulent behavior in real time during a call and takes multi-stage measures by combining it with emotion analysis, thereby making it possible to prevent fraud damage before it occurs.

[0882] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0883] Step 1:

[0884] Device: When a call is initiated, the device immediately collects audio data during the call. Audio data is captured through the device's microphone and converted into a digital format. This collected audio data is converted into text data in real time using the Google Speech-to-Text API.

[0885] Input: Audio data

[0886] Output: Text data

[0887] What it does: The microphone captures voice data, which is then sent to a speech recognition engine, which converts the speech into text and returns it to the device.

[0888] Step 2:

[0889] Terminal: The converted text data is sent to the server.

[0890] Input: Text data

[0891] Output: Text data sent to the server

[0892] Specific operation: The terminal sends the converted text data to the server via the network.

[0893] Step 3:

[0894] Server: The server analyzes the text data using natural language processing (NLP) techniques (e.g., spaCy or NLTK). The analysis process includes algorithms that detect keywords and phrases characteristic of fraud.

[0895] Input: Text data

[0896] Output: High probability of fraud detection results

[0897] What it does: It feeds text data into an NLP engine to check for the presence of fraud-related keywords, and if a fraud pattern is detected, it flags it accordingly.

[0898] Step 4:

[0899] Terminal: In parallel, the voice data during the call is also sent to an emotion engine (e.g., IBM Watson Tone Analyzer), which recognizes emotions from the user's voice.

[0900] Input: Audio data

[0901] Output: Emotion analysis results

[0902] What it does: It sends voice data to an emotion analysis engine, which analyzes the tone and patterns of the voice to identify the emotional state. If it detects emotions such as alertness or anxiety, it sends the results to a server.

[0903] Step 5:

[0904] Server: If the analysis determines that fraud is likely, the sentiment analysis is also taken into account. An enhanced alert notification is generated and sent to configured contacts.

[0905] Input: Fraud pattern detection results and sentiment analysis results

[0906] Output: Enhanced alert notifications

[0907] What it does: Based on the results of sentiment analysis, it generates prompt text to adjust the content and urgency of fraud notifications, and uses the Twilio API to send alert notifications to configured contacts.

[0908] Step 6:

[0909] Device: If a fraudulent call is deemed likely, the current call will be automatically terminated based on instructions from the server.

[0910] Input: Instructions from the server

[0911] Output: End of call

[0912] Specific operation: Upon receiving a call termination command from the server, the device forcibly terminates the call.

[0913] Step 7:

[0914] Server: If fraud is suspected, it sends a warning message to the victim's potential transaction device. This message notifies the victim of the suspected fraud.

[0915] Input: Fraud pattern detection results

[0916] Output: Warning message to trading device

[0917] What it does: If fraud is suspected, a warning message is sent via a custom API to a transaction device (e.g., an ATM), which displays the message and, if necessary, initiates additional restrictive measures.

[0918] In this way, the system of the present invention performs the necessary data processing and calculation at each step, making it possible to prevent fraud damage through multi-stage responses.

[0919] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0920] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0921] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0922] [Third embodiment]

[0923] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0924] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0925] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0926] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0927] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0928] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0929] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0930] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0931] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0932] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0933] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0934] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0935] The present invention is a system that collects voice data during a call in real time, detects signs of special fraud, and prevents fraud damage before it occurs. The specific processing flow and details of the program for this system are explained below.

[0936] 1. Collecting and converting audio data into text

[0937] Device:

[0938] When a call is initiated, the device collects real-time audio data during the call. Specifically, it uses the device's microphone to record the audio and stores it digitally. The collected audio data is immediately sent to a speech recognition engine, which converts the audio into corresponding text data. This conversion process is carried out using highly accurate speech recognition technology.

[0939] 2. Text Data Analysis

[0940] server:

[0941] The converted text data is sent to a server. The server receives the text data and analyzes it using natural language processing (NLP) techniques. This analysis includes algorithms that detect keywords and phrases characteristic of fraud. Based on the analysis results, it determines whether a fraud pattern exists.

[0942] 3. Sending alert notifications

[0943] server:

[0944] If the analysis determines that a call is likely to be fraudulent, the server immediately sends an alert notification to configured contacts (usually the victim's family or trusted individuals) warning them that a suspected fraudulent call has been detected and recommending actions to take to prevent it.

[0945] 4. Call blocking

[0946] Device:

[0947] If a fraudulent call is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the victim from further interaction with the fraudster.

[0948] 5. ATM warning messages

[0949] server:

[0950] If fraud is likely, the server will send a warning message to ATM networks that the victim may use, stating that a fraud attempt is suspected and urging caution.

[0951] ATM:

[0952] When an ATM receives a warning message, it will display a warning to the user and, if certain conditions are met (e.g., large withdrawals), will trigger additional restrictions, including withdrawal limits and additional identity verification procedures.

[0953] Specific examples

[0954] Example 1: Victim receives a call from a scammer

[0955] 1. Device: The victim's mobile phone receives a call from a scammer. The device collects the audio during the call and converts it into text in real time.

[0956] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[0957] 3. Server: Determines that the call is likely fraudulent and sends an alert to the victim's family, informing them that a suspected fraudulent call has been detected and instructing them to take immediate action.

[0958] 4. Device: Call blocking is implemented, preventing the victim from continuing the call with the scammer.

[0959] 5. Server: Sends a warning message to ATMs that the victim may use, indicating that there is a high possibility of fraud.

[0960] 6. ATMs: ATMs display warning messages to alert users to potential fraud and trigger additional restrictions based on certain conditions.

[0961] In this way, the system of the present invention can detect fraudulent activity in real time during a call and take multi-stage measures to prevent fraud damage before it occurs.

[0962] The processing flow will be explained below.

[0963] Step 1:

[0964] Device:

[0965] When a call is initiated, the device immediately collects audio data during the call, which is captured through the device's microphone and converted into a digital format.

[0966] Step 2:

[0967] Device:

[0968] The collected voice data is converted into text format in real time. A voice recognition engine is activated, processing the voice data at high speed to generate text data.

[0969] Step 3:

[0970] Device:

[0971] The converted text data is sent to the server, where it is encrypted and transferred using a secure communication protocol.

[0972] Step 4:

[0973] server:

[0974] The server receives the text data and analyzes it using a natural language processing (NLP) engine, which includes detecting keywords and phrases that may indicate fraud.

[0975] Step 5:

[0976] server:

[0977] Based on the analysis results, it is determined whether there is a possibility of fraud. If the probability of fraud exceeds a certain threshold, it is determined that there is a high possibility of fraud.

[0978] Step 6:

[0979] server:

[0980] If a fraudulent call is detected, an alert notification will be sent to your configured contacts, containing a message informing you that a suspected fraudulent call has been detected.

[0981] Step 7:

[0982] server:

[0983] After sending the alert notification, the server sends a command to the device to disconnect the call, which is also encrypted and communicated securely.

[0984] Step 8:

[0985] Device:

[0986] The device receives instructions from the server and automatically ends the current call, preventing the victim from continuing the call with the scammer.

[0987] Step 9:

[0988] server:

[0989] Assuming there is a high probability of fraud, the server will send a warning message to the affected ATM, which will prevent the victim from immediately withdrawing cash.

[0990] Step 10:

[0991] ATM:

[0992] When an ATM receives a warning message, it will display a warning accordingly, and if the user meets certain conditions (for example, if they attempt to withdraw a large amount), additional restrictions will be automatically implemented.

[0993] Example 1

[0994] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0995] Currently, the number of victims of special frauds is on the rise, and the methods targeting the elderly in particular are becoming more sophisticated. An effective system to prevent such fraud is needed, but existing systems have difficulty responding in real time, and countermeasures are often only implemented after a crime has occurred. For this reason, there is an urgent need to develop a system that can detect signs of fraud in real time during a call and respond immediately.

[0996] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0997] In this invention, the server includes means for collecting voice data during a call in real time and converting the voice into text format, means for analyzing the converted text data using natural language processing technology to detect patterns of special fraud, and means for creating prompt sentences using a generative model to evaluate the possibility of fraud. This makes it possible to detect signs of special fraud in real time during a call and immediately send an alert or block the call.

[0998] "During a call, voice data" refers to digital data of voice collected using a microphone while a user is making a call.

[0999] "Means for converting to text" refers to the speech recognition technology or software used to convert collected voice data into text.

[1000] "Natural language processing technology" is an artificial intelligence technology that analyzes text data, extracts information through an understanding of vocabulary, grammar, and meaning, and recognizes specific patterns.

[1001] "Special fraud patterns" are a collection of typical phrases, keywords, and behavioral patterns used when committing fraud.

[1002] "Means for sending alert notifications" means communications methods or technologies for sending real-time warning messages to designated contacts in the event of a high likelihood of fraud.

[1003] "Call blocking measures" are technologies or devices that forcibly terminate an ongoing call when it is determined that there is a high possibility of fraud.

[1004] "Means for sending warning messages" refers to technology for sending warning messages to automated teller machine networks that victims may use when signs of fraud are detected.

[1005] "Means for activating additional restrictions" means a function or device for imposing withdrawal limits on transactions or implementing additional identity verification procedures in accordance with certain conditions.

[1006] "Means for creating prompt sentences to assess the likelihood of fraud using a generative model" refers to a technology or method that uses an artificial intelligence model to assess the likelihood of fraud based on the frequency of occurrence of specific phrases or keywords in text data, and generates prompt sentences to present the results to users or administrators.

[1007] The present invention is a system for collecting voice data during a call in real time, detecting signs of special fraud, and preventing fraud damage before it occurs. The system of the present invention is mainly composed of terminals, a server, and an ATM.

[1008] Collecting and converting audio data into text

[1009] Device:

[1010] When a call is initiated, the device collects real-time audio data during the call. Specifically, it uses the device's microphone to record the audio and stores it digitally. The collected audio data is immediately sent to a speech recognition engine, which converts the audio into corresponding text data. This conversion process is carried out using high-precision speech recognition technology, such as Google Cloud Speech-to-Text.

[1011] Text data analysis

[1012] server:

[1013] The converted text data is sent to a server. The server receives the text data and analyzes it using natural language processing (NLP) techniques. This analysis includes generative AI models (e.g., BERT or GPT-3) to detect keywords and phrases characteristic of fraud. Based on the analysis results, it determines whether a fraud pattern exists.

[1014] Sending alert notifications

[1015] server:

[1016] If the analysis determines that a call is likely to be fraudulent, the server immediately sends an alert notification to configured contacts (usually the victim's family or trusted individuals) warning them that a suspected fraudulent call has been detected and recommending actions to take to prevent it.

[1017] Call blocking

[1018] Device:

[1019] If fraud is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the user from further interaction with the fraudster.

[1020] ATM warning message

[1021] server:

[1022] If fraud is likely, the server will send a warning message to ATM networks that the victim may use, stating that a fraud attempt is suspected and urging caution.

[1023] ATM:

[1024] When an ATM receives a warning message, it will display a warning to the user and, if certain conditions are met (e.g., large withdrawals), will trigger additional restrictions, including withdrawal limits and additional identity verification procedures.

[1025] Specific examples

[1026] Example 1: Victim receives a call from a scammer

[1027] 1. Device: Imagine a scenario where a victim's mobile phone receives a call from a scammer. The device collects the audio during the call and converts it into text in real time.

[1028] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[1029] 3. Server: Determines that the call is likely fraudulent and sends an alert notification to the victim's family, informing them that a suspected fraudulent call has been detected and providing instructions for immediate action.

[1030] 4. Device: Call blocking is performed, preventing the victim from continuing the conversation with the scammer.

[1031] 5. Server: Determines that there is a high possibility of fraud and sends a warning message to ATMs that the victim may use.

[1032] 6. ATMs: ATMs will display warning messages to notify users of potential fraud, and additional restrictions will be implemented if certain conditions are met.

[1033] Prompt Sentence Examples

[1034] Prompt: "Decide if this call is a scam"

[1035] Generative AI models analyze the frequency of specific keywords and phrases to assess the likelihood of fraud and, based on the results, recommend countermeasures in cases where fraud is suspected.

[1036] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1037] Step 1:

[1038] Start of voice data collection

[1039] Device:

[1040] When a user starts a call, the device detects the start of the call. At this time, the device's microphone becomes active and collects audio data during the call in real time. The collected audio data is saved in digital format. The input is the user's voice during the call, and the output is the collected digital audio data. Specifically, when the user presses the answer button, the device detects the start of the call using an event listener.

[1041] Step 2:

[1042] Converting audio data to text

[1043] Device:

[1044] The device sends the collected voice data to a voice recognition engine, which (for example, Google Cloud Speech-to-Text API) converts the voice into text data in real time. The input is digital voice data, and the output is the corresponding text data. Specifically, the voice recorded by the microphone is stored in digital format and sent to the voice recognition engine.

[1045] Step 3:

[1046] Sending text data

[1047] Device:

[1048] The converted text data is sent to the server. The input is text data, and the output is data transmission to the server. Specifically, the terminal sends the text data to the server using an HTTP request.

[1049] Step 4:

[1050] Text data analysis

[1051] server:

[1052] The server analyzes the received text data using natural language processing technology. This analysis uses a generative AI model that detects keywords and phrases characteristic of fraud. The input is text data, and the output is the analysis result (whether or not there is a possibility of fraud). Specifically, the server analyzes the text data using a generative AI model such as BERT or GPT-3.

[1053] Step 5:

[1054] Assessment of the likelihood of fraud

[1055] server:

[1056] The server uses a generative AI model to assess the likelihood of fraud and create a prompt. The input is analyzed text data, and the output is a prompt that assesses the likelihood of fraud. Specifically, the generative AI model evaluates specific keywords and phrases and generates a prompt based on the results.

[1057] Step 6:

[1058] Sending alert notifications

[1059] server:

[1060] If the server determines that there is a high possibility of fraud, it will send an alert notification to the configured contacts. The input is the prompt text and the alert notification destination information, and the output is sending the alert notification. Specifically, the server will send the notification using SMS or email API.

[1061] Step 7:

[1062] Call blocking

[1063] Device:

[1064] Based on instructions from the server, the terminal automatically disconnects the current call. The input is a call disconnect command from the server, and the output is the termination of the call. Specifically, the terminal uses the telephone communication module to forcibly terminate the call.

[1065] Step 8:

[1066] Sending warning messages to ATMs

[1067] server:

[1068] If the server determines that there is a high possibility of fraud, it sends a warning message to the ATM network that the victim may use. The input is a warning message and ATM network information, and the output is the transmission of the warning message. Specifically, the server sends a message to the ATM network using a network protocol.

[1069] Step 9:

[1070] ATM warning signs and restrictions

[1071] ATM:

[1072] When an ATM receives a warning message, it displays the warning to the user. Furthermore, if certain conditions are met (e.g., withdrawing a large amount), additional restrictions are implemented. The input is the received warning message and transaction conditions, and the output is the display of the warning and the implementation of restrictions. Specifically, the ATM displays the warning message on its display and, if necessary, performs additional identity verification procedures.

[1073] (Application example 1)

[1074] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1075] In recent years, there has been an increase in specialized frauds using sophisticated methods, and many people have fallen victim to them. In particular, frauds committed over the phone often target the elderly and those with little financial knowledge, so fast and effective countermeasures are needed. However, current systems lack real-time fraud detection capabilities, making it difficult to prevent damage before it occurs. For this reason, there is a need for the development of a system that can quickly detect possible fraud during a call and take appropriate countermeasures.

[1076] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1077] In this invention, the server includes means for collecting voice data during a call in real time and converting the voice into text format, means for analyzing the converted text data and detecting patterns of special fraud, means for sending an alert notification to a set contact if there is a high possibility of fraud, means for blocking the call if it is determined that there is a high possibility of fraud, means for sending a warning message to the relevant ATM, means for displaying the warning message on the user's mobile device, and means for analyzing the text data using a generative AI model. This makes it possible to detect special frauds occurring during a call in real time and take prompt measures.

[1078] "Voice data" refers to digital data of voice collected during a call using a microphone or the like of a terminal.

[1079] "Text format" is a digital data format in which voice data is converted into a string of characters.

[1080] "Converted text data" refers to data obtained by converting voice data into text format using voice recognition technology.

[1081] "Patterns of special fraud" are characteristic patterns in text data that contain keywords, phrases, and structures specific to fraudulent activity.

[1082] An "alert notification" is a warning message sent to configured contacts when potential fraud is detected.

[1083] "Call blocking measures" are features that automatically terminate an active call if there is a high likelihood of fraud.

[1084] A "warning message" is a message that warns of possible fraud.

[1085] An "ATM (Automated Teller Machine)" is an automated machine for conducting financial transactions.

[1086] "Means for displaying a warning message" refers to a function that visually displays a warning message on an ATM or a user's mobile device.

[1087] A "generative AI model" is an artificial intelligence model trained to analyze text data using natural language processing and detect fraudulent patterns.

[1088] The present invention is a system for collecting voice data during a call in real time, detecting signs of special fraud, and preventing fraud damage before it occurs. Detailed embodiments of this system are described below.

[1089] Collecting and converting audio data into text

[1090] Device:

[1091] When a call is initiated, the device uses the smartphone's microphone to collect real-time audio data during the call and stores it digitally. This audio data is then immediately sent to a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the audio into text data.

[1092] Text data analysis

[1093] server:

[1094] The converted text data is then sent to a cloud server, which receives the data and analyzes it using natural language processing (NLP) techniques, using a generative AI model (e.g., OpenAI's GPT-4) to detect keywords and phrases characteristic of fraud.

[1095] Sending alert notifications

[1096] server:

[1097] If the analysis determines that there is a high possibility of fraud, the server immediately sends an alert notification to the user's configured contacts (usually family members or trusted individuals) that warns of the suspected fraud and provides advice on how to respond.

[1098] Call blocking

[1099] Device:

[1100] If a fraudulent call is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the user from further interaction with the fraudster.

[1101] Local Warning Display

[1102] Device:

[1103] After the call is blocked, the user will receive a warning message on their mobile device, including a message about suspected fraud and next steps to take (e.g., contacting police or suspending their card).

[1104] Sending warning messages to ATMs

[1105] server:

[1106] If fraud is likely, the server will send a warning message to ATM networks that the victim may use, stating that a fraud attempt is suspected and urging caution.

[1107] ATM:

[1108] When an ATM receives a warning message, it will display a warning to the user and, if certain conditions are met (e.g., large withdrawals), will trigger additional restrictions, including withdrawal limits and additional identity verification procedures.

[1109] Specific examples

[1110] User Scenarios

[1111] 1. Call Initiation: The user receives a call from an unknown number.

[1112] 2. Speech recognition: During a call, the smartphone app converts voice into text data.

[1113] 3. NLP analysis: The text data is sent to a cloud server and analyzed using the GPT-4 model.

[1114] 4. Alerts: Keyword such as "bank account" or "transfer" are detected and an alert is sent to family members. If necessary, the call will be blocked.

[1115] 5. Local Alert: After the call ends, the user will be shown a warning message and guided on next steps.

[1116] Prompt Sentence Examples

[1117] "The text data of a suspected fraudulent call is shown below. Analyze this text data and determine whether it is likely to be fraudulent. If you are suspicious, output a warning message. Text data: 'A family member is in the hospital in an emergency. I need money immediately, so please transfer it to my bank account.'"

[1118] In this way, the present invention is a system that can detect fraudulent acts during calls in real time and prevent damage by taking multi-stage measures.

[1119] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1120] Step 1:

[1121] Collecting and converting audio data into text

[1122] Device:

[1123] When a call is initiated, the device uses a microphone to collect voice data in real time, stores the voice data in digital format, and immediately sends it to a voice recognition engine (e.g., Google Cloud Speech-to-Text API). Voice data is input and corresponding text data is output. At this time, the voice data is temporarily stored in the device's storage.

[1124] Step 2:

[1125] Sending and analyzing text data

[1126] server:

[1127] The server receives the text data sent from the device. The server sends this text data to a natural language processing (NLP) engine (e.g., OpenAI's GPT-4) to detect characteristic fraud keywords and phrases. The server receives the text data as input and outputs an analysis result indicating the likelihood of fraud.

[1128] Step 3:

[1129] Sending alert notifications

[1130] server:

[1131] If the analysis result indicates a high probability of fraud, the server will send an alert notification to the configured contacts. The alert notification will include information about the suspected fraud and the user's recommended actions. The server takes the analysis result as input and sends an alert notification to the contacts as output.

[1132] Step 4:

[1133] Call blocking

[1134] Device:

[1135] When the device receives a call-hangup command from the server, it automatically hangs up the current call, preventing the user from continuing to interact with the scammer. It takes a command from the server as input and ends the call as output.

[1136] Step 5:

[1137] Local Warning Display

[1138] Device:

[1139] After the call is blocked, the device displays a warning message on the user's screen, informing them of the suspected fraud and providing next steps. The input is the completion of the call blocking, and the output is to display a warning message on the device.

[1140] Step 6:

[1141] Sending warning messages to ATMs

[1142] server:

[1143] If the server determines that there is a particularly high possibility of fraud, it sends a warning message to the ATM network, stating that the user should exercise caution. The server takes as input data indicating a high possibility of fraud, and outputs by sending a warning message to the ATM network.

[1144] Step 7:

[1145] ATM warning message display and restriction measures

[1146] ATM:

[1147] When an ATM receives a warning message, it displays a visual warning to the user. If certain conditions are met, such as a large withdrawal, additional withdrawal restrictions or identity verification procedures are implemented. The input is a warning message from the server, and the output is the display of a warning message or the implementation of a restriction.

[1148] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1149] This invention is a system that collects voice data during calls in real time, detects signs of special fraud, and analyzes the user's emotions to prevent fraud damage before it occurs. The specific processing flow and details of the program for this system are explained below.

[1150] 1. Collecting and converting audio data into text

[1151] Device:

[1152] When a call is initiated, the device immediately collects voice data during the call. The voice data is picked up through the device's microphone and converted into digital form. The collected voice data is immediately sent to a speech recognition engine, which converts the voice into corresponding text data. This conversion process is carried out using highly accurate voice recognition technology.

[1153] 2. Text Data Analysis

[1154] server:

[1155] The converted text data is sent to a server. The server receives the text data and analyzes it using natural language processing (NLP) techniques. This analysis includes algorithms that detect keywords and phrases characteristic of fraud. Based on the analysis results, it determines whether a fraud pattern exists.

[1156] 3. Emotion analysis

[1157] Device:

[1158] Voice data during a call is also sent to the emotion engine, which recognizes emotions from the user's voice. If emotions such as alertness or anxiety are detected, that information is also sent to the server.

[1159] 4. Enhanced alert notifications

[1160] server:

[1161] If the analysis determines that a scam is likely, the results of the sentiment analysis are also taken into account. In particular, if the user's sentiment indicates vigilance or anxiety, the alert notification will be enhanced and a message urging immediate action will be generated. The server will then send the alert notification to the designated contacts (usually the victim's family or trusted individuals).

[1162] 5. Call blocking

[1163] Device:

[1164] If a fraudulent call is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the victim from further interaction with the fraudster.

[1165] 6. ATM warning messages

[1166] server:

[1167] If fraud is likely, the server will send a warning message to ATM networks that the victim may use, stating that a fraud attempt is suspected and urging caution.

[1168] ATM:

[1169] When an ATM receives a warning message, it will display a warning to the user and, if certain conditions are met (e.g., large withdrawals), will trigger additional restrictions, including withdrawal limits and additional identity verification procedures.

[1170] Specific examples

[1171] Example 1: Victim receives a call from a scammer

[1172] 1. Device: The victim's mobile phone receives a call from a scammer. The device collects the audio during the call and converts it into text in real time.

[1173] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[1174] 3. Device: At the same time, the emotion engine analyzes the user's emotions from the voice data during the call. If alertness or anxiety is detected, that information is also sent to the server.

[1175] 4. Server: Determines the likelihood of fraud and takes into account sentiment analysis results to create an enhanced alert, which includes information that a suspected fraudulent call has been detected and instructions to take immediate action.

[1176] 5. Device: Call blocking is implemented, preventing the victim from continuing the call with the scammer.

[1177] 6. Server: Sends a warning message to ATMs that the victim may use, indicating a high probability of fraud.

[1178] 7. ATMs: ATMs display warning messages to alert users to potential fraud and trigger additional restrictions based on certain conditions.

[1179] In this way, the system of the present invention detects fraudulent behavior in real time during a call and takes multi-stage measures by combining it with emotion analysis, thereby making it possible to prevent fraud damage before it occurs.

[1180] The processing flow will be explained below.

[1181] Step 1:

[1182] Device:

[1183] When a call is initiated, the device immediately begins collecting audio data during the call. It uses the device's microphone to capture audio and stores it digitally. It is set to collect audio data continuously.

[1184] Step 2:

[1185] Device:

[1186] The collected voice data is sent to a voice recognition engine in real time and instantly converted into text. The voice recognition engine uses a highly accurate recognition algorithm to generate text data from the voice. The generated text data is then stored in memory.

[1187] Step 3:

[1188] Device:

[1189] The converted text data is encrypted and sent to a server using a secure communication protocol, where it is transferred in real time to the server for analysis.

[1190] Step 4:

[1191] server:

[1192] The server receives the text data, which is then analyzed using a natural language processing (NLP) engine to detect keywords and phrases that indicate fraud.

[1193] Step 5:

[1194] server:

[1195] Based on the analysis results, we assess whether there is a possibility of fraud, using pre-defined criteria and pattern matching algorithms to determine whether there is a high probability of fraud.

[1196] Step 6:

[1197] Device:

[1198] Voice data during a call is simultaneously sent to the emotion engine, which analyzes the user's voice tone, tempo, and other factors to recognize their emotions. Emotion analysis results are generated in real time.

[1199] Step 7:

[1200] Device:

[1201] If the emotion analysis results indicate alarm or anxiety, the results are also encrypted and sent to the server. The emotion information is used as part of the fraud detection algorithm.

[1202] Step 8:

[1203] server:

[1204] If a fraudulent call is deemed likely, the results of the sentiment analysis are taken into account to enhance the alert notification, which includes a message informing users that a suspected fraudulent call has been detected and recommending immediate action.

[1205] Step 9:

[1206] server:

[1207] An alert notification is sent to configured contacts, typically family members or trusted associates of the victim, and the notification is sent immediately.

[1208] Step 10:

[1209] Device:

[1210] It receives call blocking instructions from the server and automatically ends the current call, preventing the victim from having any further contact with the scammer.

[1211] Step 11:

[1212] server:

[1213] If there is a high possibility of fraud, the server sends a warning message to the ATM, urging caution due to suspected fraud.

[1214] Step 12:

[1215] ATM:

[1216] When an ATM receives a warning message, it will display a warning accordingly. If the user meets certain conditions (e.g., withdrawing a large amount), additional restrictions will be triggered. In addition, withdrawal restrictions and additional identity verification procedures will be implemented.

[1217] In this way, the system of the present invention can detect fraudulent activity in real time during a call and take multi-stage measures to prevent fraud damage before it occurs. Furthermore, by analyzing the user's emotions, it can provide more accurate alerts and promote a quicker response.

[1218] Example 2

[1219] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1220] In today's society, where fraud damage is on the rise, there is a growing need for a system that can detect special frauds that occur during phone calls in real time and combine them with user emotion analysis to quickly and accurately prevent fraud. Conventional methods have made it difficult to detect signs of fraud in real time, making it difficult to prevent damage before it occurs. In addition, there is a demand for more accurate fraud detection by incorporating user emotion analysis.

[1221] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1222] In this invention, the server includes means for collecting voice data during a call in real time and converting the voice into text format, means for analyzing the converted text data and detecting patterns of special fraud, and means for analyzing emotions from the collected voice data in real time. This makes it possible to detect signs of special fraud occurring during a call in real time and to respond quickly and accurately based on the analysis of the user's emotions.

[1223] "Voice data during a call" refers to digital data of the voice collected through the microphone of the terminal while a call is being made.

[1224] "Real-time collection" refers to the process of capturing audio data on the fly without delay.

[1225] The "means for converting voice into text format" is a mechanism for converting voice data into corresponding character string data using voice recognition technology.

[1226] "Converted text data" is character string data generated by speech recognition technology.

[1227] "Analysis" is the process of extracting and evaluating specific information from collected data.

[1228] "Special fraud patterns" refer to specific combinations of keywords and phrases that are common to fraudulent activities.

[1229] "Emotion analysis" is the process of automatically determining a user's emotional state from speech data.

[1230] "Configured Contacts" refers to recipients of alert notifications that are pre-registered by the system.

[1231] An "alert notification" is a warning message that sends warning information to designated recipients when potential fraud is detected.

[1232] "Call interruption means" refers to the ability to interrupt and terminate communications during a call.

[1233] An "automated teller machine" is a device used for financial transactions called an ATM.

[1234] A "warning message" is a text or audio notification intended to alert a user or system.

[1235] "Specific conditions" refers to multiple criteria or rules that are set up to make the system alert.

[1236] "Additional restrictions" refers to verification procedures or functional restriction measures implemented in addition to normal operations.

[1237] This invention is a system that collects voice data during a call in real time, detects signs of special fraud, and prevents fraud damage by analyzing the user's emotions. A specific embodiment of this system is described below.

[1238] 1. Collecting and converting audio data into text

[1239] When the device detects the start of a call, it collects real-time audio data during the call through its built-in microphone. The collected audio data is converted into a digital format and sent to a speech recognition engine such as Google Cloud Speech-to-Text. The speech recognition engine converts the audio data into text data. The device then sends the converted text data to a server.

[1240] 2. Text Data Analysis

[1241] The server receives the text data sent from the device and analyzes it using natural language processing (NLP) techniques, such as SpaCy and NLTK. The server analyzes the text data to detect keywords and phrases that indicate fraud, thereby determining whether a fraud pattern exists.

[1242] 3. Emotion analysis

[1243] The device also sends voice data during the call to an emotion analysis engine, such as IBM Watson Tone Analyzer, which analyzes the user's emotions (e.g., alertness or anxiety) from the voice data and sends the results to a server. This emotion analysis result is used to evaluate signs of fraud.

[1244] 4. Enhanced alert notifications

[1245] The server evaluates the likelihood of fraud by combining the results of text analysis and sentiment analysis. If it determines that there is a high possibility of fraud, the server generates an alert notification and sends it to the specified contacts (usually the victim's family or trusted people). This alert notification will inform the user that there is a high possibility of fraud and urge them to take immediate action.

[1246] 5. Call blocking

[1247] If the server determines that there is a high possibility of fraud, the terminal will automatically disconnect the call in response to instructions from the server, thereby preventing the user from continuing the call with the fraudster.

[1248] 6. ATM warning messages

[1249] If there is a high possibility of fraud, the server sends a warning message to the ATM network that the victim may use. The ATM receives the warning message and displays a warning to the user. Furthermore, if certain conditions are met (e.g., large withdrawals), additional restrictions (withdrawal limits and identity verification procedures) are triggered.

[1250] Specific examples

[1251] Example 1: Victim receives a call from a scammer

[1252] 1. Device: The victim's mobile phone receives the call from the scammer. The device collects the audio during the call and converts it into text in real time using Google Cloud Speech-to-Text.

[1253] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[1254] 3. Device: At the same time, the call audio is analyzed for the user's emotions using IBM Watson Tone Analyzer. If alertness or anxiety is detected, that information is also sent to the server.

[1255] 4. Server: Determines the likelihood of fraud and takes into account sentiment analysis results to create an enhanced alert, informing users that a suspected fraudulent call has been detected and recommending immediate action.

[1256] 5. Device: Call blocking is implemented, preventing the victim from continuing the call with the scammer.

[1257] 6. Server: Sends a warning message to ATMs that the victim may use, indicating a high probability of fraud.

[1258] 7. ATMs: ATMs will display warning messages and trigger additional restrictions based on certain conditions.

[1259] Prompt Sentence Examples

[1260] "I want to design a system that analyzes voice data during phone calls in real time, detects signs of fraud, and enhances alert notifications based on sentiment analysis. Please explain the process flow of this system in detail."

[1261] In this way, the present invention is a system that realizes a multi-stage response to prevent fraud damage through real-time collection of voice data, text conversion, fraud detection, emotion analysis, alert notification, call blocking, and ATM warning.

[1262] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1263] Step 1:

[1264] Collecting and converting audio data into text

[1265] When the device detects the start of a call, it collects voice data during the call in real time through its built-in microphone. The input is voice data, which is converted into a digital format and sent to a voice recognition engine such as Google Cloud Speech-to-Text. The data is then processed by converting the voice data into text data using the voice recognition engine, and the output is text data.

[1266] Step 2:

[1267] Text data analysis

[1268] The server receives text data sent from the device. The input is text data, and the server analyzes this data using natural language processing (NLP) techniques such as SpaCy and NLTK. Specifically, it detects keywords and phrases that indicate fraud from the text data. Data processing involves analyzing the content of the text and identifying fraud patterns, and the output is the fraud pattern detection results.

[1269] Step 3:

[1270] Emotion analysis

[1271] The device also sends voice data during a call to an emotion analysis engine such as IBM Watson Tone Analyzer. The input is voice data, which the emotion analysis engine analyzes to determine the user's emotional state. Specifically, emotions such as vigilance and anxiety are extracted from the voice data. Data processing involves emotion analysis, and the emotion analysis results are generated as output. These results are then sent to the server.

[1272] Step 4:

[1273] Enhanced alert notifications

[1274] The server evaluates the likelihood of fraud by combining the results of text analysis and sentiment analysis. The inputs are the results of text analysis and sentiment analysis, and the server uses these to make a comprehensive judgment on the likelihood of fraud. Data calculation involves integrating the results of the two analyses to calculate the probability of fraud occurring, and the output is an alert notification message. The server then sends this alert notification to the configured contacts.

[1275] Step 5:

[1276] Call blocking

[1277] If the server determines that there is a high possibility of fraud, it sends a call-hanging instruction to the terminal. The terminal receives this instruction and automatically ends the current call. The input is the "call-hanging" instruction from the server, and the output is the call termination process. Specifically, the terminal executes the "call-hanging" command to hang up the call, preventing the user from continuing the conversation with the fraudster.

[1278] Step 6:

[1279] ATM warning message

[1280] If there is a high possibility of fraud, the server sends a warning message to the ATM network that the victim may use. The input is the possibility of fraud and related information, and the server generates a warning message based on this. Data processing involves generating a warning message, and as output, a warning message is generated that is sent to the ATM network. The ATM receives this message, displays a warning to the user, and if certain conditions are met (e.g., withdrawing a large amount), additional restrictions are triggered.

[1281] (Application example 2)

[1282] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1283] While conventional call monitoring systems can detect signs of fraud, they have the problem of being unable to analyze the user's emotions and take appropriate action based on the results. Even if signs of fraud are detected, there is a risk that the damage will increase if the user continues the call. Furthermore, even if a fraudulent call is deemed highly likely, the system lacks the functionality to immediately provide appropriate warnings and countermeasures. Therefore, a multi-stage response that takes user emotions into account is needed.

[1284] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data during a call in real time and converting the voice into text format, means for analyzing the converted text data and detecting patterns of special fraud, means for sending an alert notification to a set contact if there is a high possibility of fraud, means for cutting off the call if it is determined that there is a high possibility of fraud, means for sending a warning message to the relevant transaction device, means for analyzing the caller's emotions from the collected voice data, and means for strengthening the content of the alert notification based on the emotion analysis results. This enables multi-stage and rapid response.

[1285] "During a call, audio data" refers to audio information collected through the terminal's microphone while a call is in progress.

[1286] "Text format" refers to a format in which audio data is analyzed and converted into text data.

[1287] "Patterns of special fraud" refers to a collection of text data that includes keywords, phrases, and speaking characteristics specific to fraudulent acts.

[1288] "Configured Contacts" are emergency contacts designated by the user in advance, typically trusted family and friends.

[1289] "Alert Notification" means a warning message sent to a Contact when potential fraud is detected.

[1290] "Call blocking" is an operation that forcibly ends a current call if it is determined that there is a high possibility of fraud.

[1291] "Transaction device" refers to a device that conducts financial transactions, such as an ATM, and includes devices that have the function of receiving warning messages.

[1292] A "warning message" is a notification intended to alert a user or transaction device to suspected fraud.

[1293] "Means for analyzing the caller's emotions from collected voice data" refers to technology that uses voice data to analyze the caller's emotional state (such as vigilance or anxiety).

[1294] "Means to enhance the content of alert notifications based on the results of sentiment analysis" refers to an interface that enhances the importance and urgency of alert notifications based on sentiment analysis, encouraging appropriate action.

[1295] This invention is a system that collects voice data during calls in real time, detects signs of special fraud, and analyzes the user's emotions to prevent fraud damage before it occurs. The specific processing flow and details of the program for this system are explained below.

[1296] 1. Collecting and converting audio data into text

[1297] Device:

[1298] When a call is initiated, the device immediately collects voice data during the call. The voice data is captured through the device's microphone and converted into digital form. This collected voice data is immediately sent to a speech recognition engine (e.g., Google Speech-to-Text API) using highly accurate voice recognition technology to convert the voice into corresponding text data.

[1299] 2. Text Data Analysis

[1300] server:

[1301] The converted text data is sent to a server, which analyzes it using natural language processing (NLP) techniques (e.g., spaCy or NLTK). This analysis process includes algorithms that detect keywords and phrases characteristic of fraud. Based on the analysis results, it is determined whether a fraud pattern exists.

[1302] 3. Emotion analysis

[1303] Device:

[1304] In parallel, the voice data during the call is also sent to an emotion engine (for example, IBM Watson Tone Analyzer). The emotion engine recognizes emotions from the user's voice. If emotions such as alertness or anxiety are detected, that information is also sent to the server.

[1305] 4. Enhanced alert notifications

[1306] server:

[1307] If the analysis determines that there is a high possibility of fraud, the results of the sentiment analysis are also taken into account. If the user's sentiment indicates vigilance or anxiety, the alert notification will be enhanced and a message urging immediate action will be generated. The server will then send the alert notification to the configured contacts (usually the victim's family or trusted individuals). The notification can be sent using the Twilio API.

[1308] 5. Call blocking

[1309] Device:

[1310] If a fraudulent call is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the victim from further interaction with the fraudster.

[1311] 6. ATM warning messages

[1312] server:

[1313] If fraud is likely, the server will send a warning message to transaction devices that the victim may use (e.g., ATM networks), stating that a fraud is suspected and urging them to be careful.

[1314] Trading Device:

[1315] Upon receiving the warning message, the transaction device will display a warning to the user, and if certain conditions are met (e.g., large withdrawals), additional restrictions may be imposed, including withdrawal limits and additional identity verification procedures.

[1316] Specific examples

[1317] Example 1: Victim receives a call from a scammer

[1318] 1. Device: The victim's mobile phone receives a call from a scammer. The device collects the audio during the call and converts it into text in real time.

[1319] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[1320] 3. Device: At the same time, the emotion engine analyzes the user's emotions from the voice data during the call. If alertness or anxiety is detected, that information is also sent to the server.

[1321] 4. Server: Determines the likelihood of fraud and takes into account sentiment analysis results to create an enhanced alert, which includes information that a suspected fraudulent call has been detected and instructions to take immediate action.

[1322] 5. Device: Call blocking is implemented, preventing the victim from continuing the call with the scammer.

[1323] 6. Server: Sends a warning message to transaction devices that the victim may use, indicating that there is a high possibility of fraud.

[1324] 7. Trading Device: The trading device displays warning messages, notifies users of potential fraud, and triggers additional restrictions based on certain conditions.

[1325] Example prompt sentence:

[1326] "This call shows signs of fraud. It contains the following phrases: bank account, large withdrawal, family emergency. The scam was also confirmed through sentiment analysis. End the call immediately and send an alert notification to a trusted contact."

[1327] In this way, the system of the present invention detects fraudulent behavior in real time during a call and takes multi-stage measures by combining it with emotion analysis, thereby making it possible to prevent fraud damage before it occurs.

[1328] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1329] Step 1:

[1330] Device: When a call is initiated, the device immediately collects audio data during the call. Audio data is captured through the device's microphone and converted into a digital format. This collected audio data is converted into text data in real time using the Google Speech-to-Text API.

[1331] Input: Audio data

[1332] Output: Text data

[1333] What it does: The microphone captures voice data, which is then sent to a speech recognition engine, which converts the speech into text and returns it to the device.

[1334] Step 2:

[1335] Terminal: The converted text data is sent to the server.

[1336] Input: Text data

[1337] Output: Text data sent to the server

[1338] Specific operation: The terminal sends the converted text data to the server via the network.

[1339] Step 3:

[1340] Server: The server analyzes the text data using natural language processing (NLP) techniques (e.g., spaCy or NLTK). The analysis process includes algorithms that detect keywords and phrases characteristic of fraud.

[1341] Input: Text data

[1342] Output: High probability of fraud detection results

[1343] What it does: It feeds text data into an NLP engine to check for the presence of fraud-related keywords, and if a fraud pattern is detected, it flags it accordingly.

[1344] Step 4:

[1345] Terminal: In parallel, the voice data during the call is also sent to an emotion engine (e.g., IBM Watson Tone Analyzer), which recognizes emotions from the user's voice.

[1346] Input: Audio data

[1347] Output: Emotion analysis results

[1348] What it does: It sends voice data to an emotion analysis engine, which analyzes the tone and patterns of the voice to identify the emotional state. If it detects emotions such as alertness or anxiety, it sends the results to a server.

[1349] Step 5:

[1350] Server: If the analysis determines that fraud is likely, the sentiment analysis is also taken into account. An enhanced alert notification is generated and sent to configured contacts.

[1351] Input: Fraud pattern detection results and sentiment analysis results

[1352] Output: Enhanced alert notifications

[1353] What it does: Based on the results of sentiment analysis, it generates prompt text to adjust the content and urgency of fraud notifications, and uses the Twilio API to send alert notifications to configured contacts.

[1354] Step 6:

[1355] Device: If a fraudulent call is deemed likely, the current call will be automatically terminated based on instructions from the server.

[1356] Input: Instructions from the server

[1357] Output: End of call

[1358] Specific operation: Upon receiving a call termination command from the server, the device forcibly terminates the call.

[1359] Step 7:

[1360] Server: If fraud is suspected, it sends a warning message to the victim's potential transaction device. This message notifies the victim of the suspected fraud.

[1361] Input: Fraud pattern detection results

[1362] Output: Warning message to trading device

[1363] What it does: If fraud is suspected, a warning message is sent via a custom API to a transaction device (e.g., an ATM), which displays the message and, if necessary, initiates additional restrictive measures.

[1364] In this way, the system of the present invention performs the necessary data processing and calculation at each step, making it possible to prevent fraud damage through multi-stage responses.

[1365] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1366] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1367] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1368] [Fourth embodiment]

[1369] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1370] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1371] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1372] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1373] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1374] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1375] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1376] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1377] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1378] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1379] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1380] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1381] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1382] The present invention is a system that collects voice data during a call in real time, detects signs of special fraud, and prevents fraud damage before it occurs. The specific processing flow and details of the program for this system are explained below.

[1383] 1. Collecting and converting audio data into text

[1384] Device:

[1385] When a call is initiated, the device collects real-time audio data during the call. Specifically, it uses the device's microphone to record the audio and stores it digitally. The collected audio data is immediately sent to a speech recognition engine, which converts the audio into corresponding text data. This conversion process is carried out using highly accurate speech recognition technology.

[1386] 2. Text Data Analysis

[1387] server:

[1388] The converted text data is sent to a server. The server receives the text data and analyzes it using natural language processing (NLP) techniques. This analysis includes algorithms that detect keywords and phrases characteristic of fraud. Based on the analysis results, it determines whether a fraud pattern exists.

[1389] 3. Sending alert notifications

[1390] server:

[1391] If the analysis determines that a call is likely to be fraudulent, the server immediately sends an alert notification to configured contacts (usually the victim's family or trusted individuals) warning them that a suspected fraudulent call has been detected and recommending actions to take to prevent it.

[1392] 4. Call blocking

[1393] Device:

[1394] If a fraudulent call is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the victim from further interaction with the fraudster.

[1395] 5. ATM warning messages

[1396] server:

[1397] If fraud is likely, the server will send a warning message to ATM networks that the victim may use, stating that a fraud attempt is suspected and urging caution.

[1398] ATM:

[1399] When an ATM receives a warning message, it will display a warning to the user and, if certain conditions are met (e.g., large withdrawals), will trigger additional restrictions, including withdrawal limits and additional identity verification procedures.

[1400] Specific examples

[1401] Example 1: Victim receives a call from a scammer

[1402] 1. Device: The victim's mobile phone receives a call from a scammer. The device collects the audio during the call and converts it into text in real time.

[1403] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[1404] 3. Server: Determines that the call is likely fraudulent and sends an alert to the victim's family, informing them that a suspected fraudulent call has been detected and instructing them to take immediate action.

[1405] 4. Device: Call blocking is implemented, preventing the victim from continuing the call with the scammer.

[1406] 5. Server: Sends a warning message to ATMs that the victim may use, indicating that there is a high possibility of fraud.

[1407] 6. ATMs: ATMs display warning messages to alert users to potential fraud and trigger additional restrictions based on certain conditions.

[1408] In this way, the system of the present invention can detect fraudulent activity in real time during a call and take multi-stage measures to prevent fraud damage before it occurs.

[1409] The processing flow will be explained below.

[1410] Step 1:

[1411] Device:

[1412] When a call is initiated, the device immediately collects audio data during the call, which is captured through the device's microphone and converted into a digital format.

[1413] Step 2:

[1414] Device:

[1415] The collected voice data is converted into text format in real time. A voice recognition engine is activated, processing the voice data at high speed to generate text data.

[1416] Step 3:

[1417] Device:

[1418] The converted text data is sent to the server, where it is encrypted and transferred using a secure communication protocol.

[1419] Step 4:

[1420] server:

[1421] The server receives the text data and analyzes it using a natural language processing (NLP) engine, which includes detecting keywords and phrases that may indicate fraud.

[1422] Step 5:

[1423] server:

[1424] Based on the analysis results, it is determined whether there is a possibility of fraud. If the probability of fraud exceeds a certain threshold, it is determined that there is a high possibility of fraud.

[1425] Step 6:

[1426] server:

[1427] If a fraudulent call is detected, an alert notification will be sent to your configured contacts, containing a message informing you that a suspected fraudulent call has been detected.

[1428] Step 7:

[1429] server:

[1430] After sending the alert notification, the server sends a command to the device to disconnect the call, which is also encrypted and communicated securely.

[1431] Step 8:

[1432] Device:

[1433] The device receives instructions from the server and automatically ends the current call, preventing the victim from continuing the call with the scammer.

[1434] Step 9:

[1435] server:

[1436] Assuming there is a high probability of fraud, the server will send a warning message to the affected ATM, which will prevent the victim from immediately withdrawing cash.

[1437] Step 10:

[1438] ATM:

[1439] When an ATM receives a warning message, it will display a warning accordingly, and if the user meets certain conditions (for example, if they attempt to withdraw a large amount), additional restrictions will be automatically implemented.

[1440] Example 1

[1441] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1442] Currently, the number of victims of special frauds is on the rise, and the methods targeting the elderly in particular are becoming more sophisticated. An effective system to prevent such fraud is needed, but existing systems have difficulty responding in real time, and countermeasures are often only implemented after a crime has occurred. For this reason, there is an urgent need to develop a system that can detect signs of fraud in real time during a call and respond immediately.

[1443] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1444] In this invention, the server includes means for collecting voice data during a call in real time and converting the voice into text format, means for analyzing the converted text data using natural language processing technology to detect patterns of special fraud, and means for creating prompt sentences using a generative model to evaluate the possibility of fraud. This makes it possible to detect signs of special fraud in real time during a call and immediately send an alert or block the call.

[1445] "During a call, voice data" refers to digital data of voice collected using a microphone while a user is making a call.

[1446] "Means for converting to text" refers to the speech recognition technology or software used to convert collected voice data into text.

[1447] "Natural language processing technology" is an artificial intelligence technology that analyzes text data, extracts information through an understanding of vocabulary, grammar, and meaning, and recognizes specific patterns.

[1448] "Special fraud patterns" are a collection of typical phrases, keywords, and behavioral patterns used when committing fraud.

[1449] "Means for sending alert notifications" means communications methods or technologies for sending real-time warning messages to designated contacts in the event of a high likelihood of fraud.

[1450] "Call blocking measures" are technologies or devices that forcibly terminate an ongoing call when it is determined that there is a high possibility of fraud.

[1451] "Means for sending warning messages" refers to technology for sending warning messages to automated teller machine networks that victims may use when signs of fraud are detected.

[1452] "Means for activating additional restrictions" means a function or device for imposing withdrawal limits on transactions or implementing additional identity verification procedures in accordance with certain conditions.

[1453] "Means for creating prompt sentences to assess the likelihood of fraud using a generative model" refers to a technology or method that uses an artificial intelligence model to assess the likelihood of fraud based on the frequency of occurrence of specific phrases or keywords in text data, and generates prompt sentences to present the results to users or administrators.

[1454] The present invention is a system for collecting voice data during a call in real time, detecting signs of special fraud, and preventing fraud damage before it occurs. The system of the present invention is mainly composed of terminals, a server, and an ATM.

[1455] Collecting and converting audio data into text

[1456] Device:

[1457] When a call is initiated, the device collects real-time audio data during the call. Specifically, it uses the device's microphone to record the audio and stores it digitally. The collected audio data is immediately sent to a speech recognition engine, which converts the audio into corresponding text data. This conversion process is carried out using high-precision speech recognition technology, such as Google Cloud Speech-to-Text.

[1458] Text data analysis

[1459] server:

[1460] The converted text data is sent to a server. The server receives the text data and analyzes it using natural language processing (NLP) techniques. This analysis includes generative AI models (e.g., BERT or GPT-3) to detect keywords and phrases characteristic of fraud. Based on the analysis results, it determines whether a fraud pattern exists.

[1461] Sending alert notifications

[1462] server:

[1463] If the analysis determines that a call is likely to be fraudulent, the server immediately sends an alert notification to configured contacts (usually the victim's family or trusted individuals) warning them that a suspected fraudulent call has been detected and recommending actions to take to prevent it.

[1464] Call blocking

[1465] Device:

[1466] If fraud is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the user from further interaction with the fraudster.

[1467] ATM warning message

[1468] server:

[1469] If fraud is likely, the server will send a warning message to ATM networks that the victim may use, stating that a fraud attempt is suspected and urging caution.

[1470] ATM:

[1471] When an ATM receives a warning message, it will display a warning to the user and, if certain conditions are met (e.g., large withdrawals), will trigger additional restrictions, including withdrawal limits and additional identity verification procedures.

[1472] Specific examples

[1473] Example 1: Victim receives a call from a scammer

[1474] 1. Device: Imagine a scenario where a victim's mobile phone receives a call from a scammer. The device collects the audio during the call and converts it into text in real time.

[1475] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[1476] 3. Server: Determines that the call is likely fraudulent and sends an alert notification to the victim's family, informing them that a suspected fraudulent call has been detected and providing instructions for immediate action.

[1477] 4. Device: Call blocking is performed, preventing the victim from continuing the conversation with the scammer.

[1478] 5. Server: Determines that there is a high possibility of fraud and sends a warning message to ATMs that the victim may use.

[1479] 6. ATMs: ATMs will display warning messages to notify users of potential fraud, and additional restrictions will be implemented if certain conditions are met.

[1480] Prompt Sentence Examples

[1481] Prompt: "Decide if this call is a scam"

[1482] Generative AI models analyze the frequency of specific keywords and phrases to assess the likelihood of fraud and, based on the results, recommend countermeasures in cases where fraud is suspected.

[1483] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1484] Step 1:

[1485] Start of voice data collection

[1486] Device:

[1487] When a user starts a call, the device detects the start of the call. At this time, the device's microphone becomes active and collects audio data during the call in real time. The collected audio data is saved in digital format. The input is the user's voice during the call, and the output is the collected digital audio data. Specifically, when the user presses the answer button, the device detects the start of the call using an event listener.

[1488] Step 2:

[1489] Converting audio data to text

[1490] Device:

[1491] The device sends the collected voice data to a voice recognition engine, which (for example, Google Cloud Speech-to-Text API) converts the voice into text data in real time. The input is digital voice data, and the output is the corresponding text data. Specifically, the voice recorded by the microphone is stored in digital format and sent to the voice recognition engine.

[1492] Step 3:

[1493] Sending text data

[1494] Device:

[1495] The converted text data is sent to the server. The input is text data, and the output is data transmission to the server. Specifically, the terminal sends the text data to the server using an HTTP request.

[1496] Step 4:

[1497] Text data analysis

[1498] server:

[1499] The server analyzes the received text data using natural language processing technology. This analysis uses a generative AI model that detects keywords and phrases characteristic of fraud. The input is text data, and the output is the analysis result (whether or not there is a possibility of fraud). Specifically, the server analyzes the text data using a generative AI model such as BERT or GPT-3.

[1500] Step 5:

[1501] Assessment of the likelihood of fraud

[1502] server:

[1503] The server uses a generative AI model to assess the likelihood of fraud and create a prompt. The input is analyzed text data, and the output is a prompt that assesses the likelihood of fraud. Specifically, the generative AI model evaluates specific keywords and phrases and generates a prompt based on the results.

[1504] Step 6:

[1505] Sending alert notifications

[1506] server:

[1507] If the server determines that there is a high possibility of fraud, it will send an alert notification to the configured contacts. The input is the prompt text and the alert notification destination information, and the output is sending the alert notification. Specifically, the server will send the notification using SMS or email API.

[1508] Step 7:

[1509] Call blocking

[1510] Device:

[1511] Based on instructions from the server, the terminal automatically disconnects the current call. The input is a call disconnect command from the server, and the output is the termination of the call. Specifically, the terminal uses the telephone communication module to forcibly terminate the call.

[1512] Step 8:

[1513] Sending warning messages to ATMs

[1514] server:

[1515] If the server determines that there is a high possibility of fraud, it sends a warning message to the ATM network that the victim may use. The input is a warning message and ATM network information, and the output is the transmission of the warning message. Specifically, the server sends a message to the ATM network using a network protocol.

[1516] Step 9:

[1517] ATM warning signs and restrictions

[1518] ATM:

[1519] When an ATM receives a warning message, it displays the warning to the user. Furthermore, if certain conditions are met (e.g., withdrawing a large amount), additional restrictions are implemented. The input is the received warning message and transaction conditions, and the output is the display of the warning and the implementation of restrictions. Specifically, the ATM displays the warning message on its display and, if necessary, performs additional identity verification procedures.

[1520] (Application example 1)

[1521] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1522] In recent years, there has been an increase in specialized frauds using sophisticated methods, and many people have fallen victim to them. In particular, frauds committed over the phone often target the elderly and those with little financial knowledge, so fast and effective countermeasures are needed. However, current systems lack real-time fraud detection capabilities, making it difficult to prevent damage before it occurs. For this reason, there is a need for the development of a system that can quickly detect possible fraud during a call and take appropriate countermeasures.

[1523] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1524] In this invention, the server includes means for collecting voice data during a call in real time and converting the voice into text format, means for analyzing the converted text data and detecting patterns of special fraud, means for sending an alert notification to a set contact if there is a high possibility of fraud, means for blocking the call if it is determined that there is a high possibility of fraud, means for sending a warning message to the relevant ATM, means for displaying the warning message on the user's mobile device, and means for analyzing the text data using a generative AI model. This makes it possible to detect special frauds occurring during a call in real time and take prompt measures.

[1525] "Voice data" refers to digital data of voice collected during a call using a microphone or the like of a terminal.

[1526] "Text format" is a digital data format in which voice data is converted into a string of characters.

[1527] "Converted text data" refers to data obtained by converting voice data into text format using voice recognition technology.

[1528] "Patterns of special fraud" are characteristic patterns in text data that contain keywords, phrases, and structures specific to fraudulent activity.

[1529] An "alert notification" is a warning message sent to configured contacts when potential fraud is detected.

[1530] "Call blocking measures" are features that automatically terminate an active call if there is a high likelihood of fraud.

[1531] A "warning message" is a message that warns of possible fraud.

[1532] An "ATM (Automated Teller Machine)" is an automated machine for conducting financial transactions.

[1533] "Means for displaying a warning message" refers to a function that visually displays a warning message on an ATM or a user's mobile device.

[1534] A "generative AI model" is an artificial intelligence model trained to analyze text data using natural language processing and detect fraudulent patterns.

[1535] The present invention is a system for collecting voice data during a call in real time, detecting signs of special fraud, and preventing fraud damage before it occurs. Detailed embodiments of this system are described below.

[1536] Collecting and converting audio data into text

[1537] Device:

[1538] When a call is initiated, the device uses the smartphone's microphone to collect real-time audio data during the call and stores it digitally. This audio data is then immediately sent to a speech recognition engine (e.g., Google Cloud Speech-to-Text API) to convert the audio into text data.

[1539] Text data analysis

[1540] server:

[1541] The converted text data is then sent to a cloud server, which receives the data and analyzes it using natural language processing (NLP) techniques, using a generative AI model (e.g., OpenAI's GPT-4) to detect keywords and phrases characteristic of fraud.

[1542] Sending alert notifications

[1543] server:

[1544] If the analysis determines that there is a high possibility of fraud, the server immediately sends an alert notification to the user's configured contacts (usually family members or trusted individuals) that warns of the suspected fraud and provides advice on how to respond.

[1545] Call blocking

[1546] Device:

[1547] If a fraudulent call is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the user from further interaction with the fraudster.

[1548] Local Warning Display

[1549] Device:

[1550] After the call is blocked, the user will receive a warning message on their mobile device, including a message about suspected fraud and next steps to take (e.g., contacting police or suspending their card).

[1551] Sending warning messages to ATMs

[1552] server:

[1553] If fraud is likely, the server will send a warning message to ATM networks that the victim may use, stating that a fraud attempt is suspected and urging caution.

[1554] ATM:

[1555] When an ATM receives a warning message, it will display a warning to the user and, if certain conditions are met (e.g., large withdrawals), will trigger additional restrictions, including withdrawal limits and additional identity verification procedures.

[1556] Specific examples

[1557] User Scenarios

[1558] 1. Call Initiation: The user receives a call from an unknown number.

[1559] 2. Speech recognition: During a call, the smartphone app converts voice into text data.

[1560] 3. NLP analysis: The text data is sent to a cloud server and analyzed using the GPT-4 model.

[1561] 4. Alerts: Keyword such as "bank account" or "transfer" are detected and an alert is sent to family members. If necessary, the call will be blocked.

[1562] 5. Local Alert: After the call ends, the user will be shown a warning message and guided on next steps.

[1563] Prompt Sentence Examples

[1564] "The text data of a suspected fraudulent call is shown below. Analyze this text data and determine whether it is likely to be fraudulent. If you are suspicious, output a warning message. Text data: 'A family member is in the hospital in an emergency. I need money immediately, so please transfer it to my bank account.'"

[1565] In this way, the present invention is a system that can detect fraudulent acts during calls in real time and prevent damage by taking multi-stage measures.

[1566] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1567] Step 1:

[1568] Collecting and converting audio data into text

[1569] Device:

[1570] When a call is initiated, the device uses a microphone to collect voice data in real time, stores the voice data in digital format, and immediately sends it to a voice recognition engine (e.g., Google Cloud Speech-to-Text API). Voice data is input and corresponding text data is output. At this time, the voice data is temporarily stored in the device's storage.

[1571] Step 2:

[1572] Sending and analyzing text data

[1573] server:

[1574] The server receives the text data sent from the device. The server sends this text data to a natural language processing (NLP) engine (e.g., OpenAI's GPT-4) to detect characteristic fraud keywords and phrases. The server receives the text data as input and outputs an analysis result indicating the likelihood of fraud.

[1575] Step 3:

[1576] Sending alert notifications

[1577] server:

[1578] If the analysis result indicates a high probability of fraud, the server will send an alert notification to the configured contacts. The alert notification will include information about the suspected fraud and the user's recommended actions. The server takes the analysis result as input and sends an alert notification to the contacts as output.

[1579] Step 4:

[1580] Call blocking

[1581] Device:

[1582] When the device receives a call-hangup command from the server, it automatically hangs up the current call, preventing the user from continuing to interact with the scammer. It takes a command from the server as input and ends the call as output.

[1583] Step 5:

[1584] Local Warning Display

[1585] Device:

[1586] After the call is blocked, the device displays a warning message on the user's screen, informing them of the suspected fraud and providing next steps. The input is the completion of the call blocking, and the output is to display a warning message on the device.

[1587] Step 6:

[1588] Sending warning messages to ATMs

[1589] server:

[1590] If the server determines that there is a particularly high possibility of fraud, it sends a warning message to the ATM network, stating that the user should exercise caution. The server takes as input data indicating a high possibility of fraud, and outputs by sending a warning message to the ATM network.

[1591] Step 7:

[1592] ATM warning message display and restriction measures

[1593] ATM:

[1594] When an ATM receives a warning message, it displays a visual warning to the user. If certain conditions are met, such as a large withdrawal, additional withdrawal restrictions or identity verification procedures are implemented. The input is a warning message from the server, and the output is the display of a warning message or the implementation of a restriction.

[1595] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1596] This invention is a system that collects voice data during calls in real time, detects signs of special fraud, and analyzes the user's emotions to prevent fraud damage before it occurs. The specific processing flow and details of the program for this system are explained below.

[1597] 1. Collecting and converting audio data into text

[1598] Device:

[1599] When a call is initiated, the device immediately collects voice data during the call. The voice data is picked up through the device's microphone and converted into digital form. The collected voice data is immediately sent to a speech recognition engine, which converts the voice into corresponding text data. This conversion process is carried out using highly accurate voice recognition technology.

[1600] 2. Text Data Analysis

[1601] server:

[1602] The converted text data is sent to a server. The server receives the text data and analyzes it using natural language processing (NLP) techniques. This analysis includes algorithms that detect keywords and phrases characteristic of fraud. Based on the analysis results, it determines whether a fraud pattern exists.

[1603] 3. Emotion analysis

[1604] Device:

[1605] Voice data during a call is also sent to the emotion engine, which recognizes emotions from the user's voice. If emotions such as alertness or anxiety are detected, that information is also sent to the server.

[1606] 4. Enhanced alert notifications

[1607] server:

[1608] If the analysis determines that a scam is likely, the results of the sentiment analysis are also taken into account. In particular, if the user's sentiment indicates vigilance or anxiety, the alert notification will be enhanced and a message urging immediate action will be generated. The server will then send the alert notification to the designated contacts (usually the victim's family or trusted individuals).

[1609] 5. Call blocking

[1610] Device:

[1611] If a fraudulent call is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the victim from further interaction with the fraudster.

[1612] 6. ATM warning messages

[1613] server:

[1614] If fraud is likely, the server will send a warning message to ATM networks that the victim may use, stating that a fraud attempt is suspected and urging caution.

[1615] ATM:

[1616] When an ATM receives a warning message, it will display a warning to the user and, if certain conditions are met (e.g., large withdrawals), will trigger additional restrictions, including withdrawal limits and additional identity verification procedures.

[1617] Specific examples

[1618] Example 1: Victim receives a call from a scammer

[1619] 1. Device: The victim's mobile phone receives a call from a scammer. The device collects the audio during the call and converts it into text in real time.

[1620] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[1621] 3. Device: At the same time, the emotion engine analyzes the user's emotions from the voice data during the call. If alertness or anxiety is detected, that information is also sent to the server.

[1622] 4. Server: Determines the likelihood of fraud and takes into account sentiment analysis results to create an enhanced alert, which includes information that a suspected fraudulent call has been detected and instructions to take immediate action.

[1623] 5. Device: Call blocking is implemented, preventing the victim from continuing the call with the scammer.

[1624] 6. Server: Sends a warning message to ATMs that the victim may use, indicating a high probability of fraud.

[1625] 7. ATMs: ATMs display warning messages to alert users to potential fraud and trigger additional restrictions based on certain conditions.

[1626] In this way, the system of the present invention detects fraudulent behavior in real time during a call and takes multi-stage measures by combining it with emotion analysis, thereby making it possible to prevent fraud damage before it occurs.

[1627] The processing flow will be explained below.

[1628] Step 1:

[1629] Device:

[1630] When a call is initiated, the device immediately begins collecting audio data during the call. It uses the device's microphone to capture audio and stores it digitally. It is set to collect audio data continuously.

[1631] Step 2:

[1632] Device:

[1633] The collected voice data is sent to a voice recognition engine in real time and instantly converted into text. The voice recognition engine uses a highly accurate recognition algorithm to generate text data from the voice. The generated text data is then stored in memory.

[1634] Step 3:

[1635] Device:

[1636] The converted text data is encrypted and sent to a server using a secure communication protocol, where it is transferred in real time to the server for analysis.

[1637] Step 4:

[1638] server:

[1639] The server receives the text data, which is then analyzed using a natural language processing (NLP) engine to detect keywords and phrases that indicate fraud.

[1640] Step 5:

[1641] server:

[1642] Based on the analysis results, we assess whether there is a possibility of fraud, using pre-defined criteria and pattern matching algorithms to determine whether there is a high probability of fraud.

[1643] Step 6:

[1644] Device:

[1645] Voice data during a call is simultaneously sent to the emotion engine, which analyzes the user's voice tone, tempo, and other factors to recognize their emotions. Emotion analysis results are generated in real time.

[1646] Step 7:

[1647] Device:

[1648] If the emotion analysis results indicate alarm or anxiety, the results are also encrypted and sent to the server. The emotion information is used as part of the fraud detection algorithm.

[1649] Step 8:

[1650] server:

[1651] If a fraudulent call is deemed likely, the results of the sentiment analysis are taken into account to enhance the alert notification, which includes a message informing users that a suspected fraudulent call has been detected and recommending immediate action.

[1652] Step 9:

[1653] server:

[1654] An alert notification is sent to configured contacts, typically family members or trusted associates of the victim, and the notification is sent immediately.

[1655] Step 10:

[1656] Device:

[1657] It receives call blocking instructions from the server and automatically ends the current call, preventing the victim from having any further contact with the scammer.

[1658] Step 11:

[1659] server:

[1660] If there is a high possibility of fraud, the server sends a warning message to the ATM, urging caution due to suspected fraud.

[1661] Step 12:

[1662] ATM:

[1663] When an ATM receives a warning message, it will display a warning accordingly. If the user meets certain conditions (e.g., withdrawing a large amount), additional restrictions will be triggered. In addition, withdrawal restrictions and additional identity verification procedures will be implemented.

[1664] In this way, the system of the present invention can detect fraudulent activity in real time during a call and take multi-stage measures to prevent fraud damage before it occurs. Furthermore, by analyzing the user's emotions, it can provide more accurate alerts and promote a quicker response.

[1665] Example 2

[1666] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1667] In today's society, where fraud damage is on the rise, there is a growing need for a system that can detect special frauds that occur during phone calls in real time and combine them with user emotion analysis to quickly and accurately prevent fraud. Conventional methods have made it difficult to detect signs of fraud in real time, making it difficult to prevent damage before it occurs. In addition, there is a demand for more accurate fraud detection by incorporating user emotion analysis.

[1668] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1669] In this invention, the server includes means for collecting voice data during a call in real time and converting the voice into text format, means for analyzing the converted text data and detecting patterns of special fraud, and means for analyzing emotions from the collected voice data in real time. This makes it possible to detect signs of special fraud occurring during a call in real time and to respond quickly and accurately based on the analysis of the user's emotions.

[1670] "Voice data during a call" refers to digital data of the voice collected through the microphone of the terminal while a call is being made.

[1671] "Real-time collection" refers to the process of capturing audio data on the fly without delay.

[1672] The "means for converting voice into text format" is a mechanism for converting voice data into corresponding character string data using voice recognition technology.

[1673] "Converted text data" is character string data generated by speech recognition technology.

[1674] "Analysis" is the process of extracting and evaluating specific information from collected data.

[1675] "Special fraud patterns" refer to specific combinations of keywords and phrases that are common to fraudulent activities.

[1676] "Emotion analysis" is the process of automatically determining a user's emotional state from speech data.

[1677] "Configured Contacts" refers to recipients of alert notifications that are pre-registered by the system.

[1678] An "alert notification" is a warning message that sends warning information to designated recipients when potential fraud is detected.

[1679] "Call interruption means" refers to the ability to interrupt and terminate communications during a call.

[1680] An "automated teller machine" is a device used for financial transactions called an ATM.

[1681] A "warning message" is a text or audio notification intended to alert a user or system.

[1682] "Specific conditions" refers to multiple criteria or rules that are set up to make the system alert.

[1683] "Additional restrictions" refers to verification procedures or functional restriction measures implemented in addition to normal operations.

[1684] This invention is a system that collects voice data during a call in real time, detects signs of special fraud, and prevents fraud damage by analyzing the user's emotions. A specific embodiment of this system is described below.

[1685] 1. Collecting and converting audio data into text

[1686] When the device detects the start of a call, it collects real-time audio data during the call through its built-in microphone. The collected audio data is converted into a digital format and sent to a speech recognition engine such as Google Cloud Speech-to-Text. The speech recognition engine converts the audio data into text data. The device then sends the converted text data to a server.

[1687] 2. Text Data Analysis

[1688] The server receives the text data sent from the device and analyzes it using natural language processing (NLP) techniques, such as SpaCy and NLTK. The server analyzes the text data to detect keywords and phrases that indicate fraud, thereby determining whether a fraud pattern exists.

[1689] 3. Emotion analysis

[1690] The device also sends voice data during the call to an emotion analysis engine, such as IBM Watson Tone Analyzer, which analyzes the user's emotions (e.g., alertness or anxiety) from the voice data and sends the results to a server. This emotion analysis result is used to evaluate signs of fraud.

[1691] 4. Enhanced alert notifications

[1692] The server evaluates the likelihood of fraud by combining the results of text analysis and sentiment analysis. If it determines that there is a high possibility of fraud, the server generates an alert notification and sends it to the specified contacts (usually the victim's family or trusted people). This alert notification will inform the user that there is a high possibility of fraud and urge them to take immediate action.

[1693] 5. Call blocking

[1694] If the server determines that there is a high possibility of fraud, the terminal will automatically disconnect the call in response to instructions from the server, thereby preventing the user from continuing the call with the fraudster.

[1695] 6. ATM warning messages

[1696] If there is a high possibility of fraud, the server sends a warning message to the ATM network that the victim may use. The ATM receives the warning message and displays a warning to the user. Furthermore, if certain conditions are met (e.g., large withdrawals), additional restrictions (withdrawal limits and identity verification procedures) are triggered.

[1697] Specific examples

[1698] Example 1: Victim receives a call from a scammer

[1699] 1. Device: The victim's mobile phone receives the call from the scammer. The device collects the audio during the call and converts it into text in real time using Google Cloud Speech-to-Text.

[1700] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[1701] 3. Device: At the same time, the call audio is analyzed for the user's emotions using IBM Watson Tone Analyzer. If alertness or anxiety is detected, that information is also sent to the server.

[1702] 4. Server: Determines the likelihood of fraud and takes into account sentiment analysis results to create an enhanced alert, informing users that a suspected fraudulent call has been detected and recommending immediate action.

[1703] 5. Device: Call blocking is implemented, preventing the victim from continuing the call with the scammer.

[1704] 6. Server: Sends a warning message to ATMs that the victim may use, indicating a high probability of fraud.

[1705] 7. ATMs: ATMs will display warning messages and trigger additional restrictions based on certain conditions.

[1706] Prompt Sentence Examples

[1707] "I want to design a system that analyzes voice data during phone calls in real time, detects signs of fraud, and enhances alert notifications based on sentiment analysis. Please explain the process flow of this system in detail."

[1708] In this way, the present invention is a system that realizes a multi-stage response to prevent fraud damage through real-time collection of voice data, text conversion, fraud detection, emotion analysis, alert notification, call blocking, and ATM warning.

[1709] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1710] Step 1:

[1711] Collecting and converting audio data into text

[1712] When the device detects the start of a call, it collects voice data during the call in real time through its built-in microphone. The input is voice data, which is converted into a digital format and sent to a voice recognition engine such as Google Cloud Speech-to-Text. The data is then processed by converting the voice data into text data using the voice recognition engine, and the output is text data.

[1713] Step 2:

[1714] Text data analysis

[1715] The server receives text data sent from the device. The input is text data, and the server analyzes this data using natural language processing (NLP) techniques such as SpaCy and NLTK. Specifically, it detects keywords and phrases that indicate fraud from the text data. Data processing involves analyzing the content of the text and identifying fraud patterns, and the output is the fraud pattern detection results.

[1716] Step 3:

[1717] Emotion analysis

[1718] The device also sends voice data during a call to an emotion analysis engine such as IBM Watson Tone Analyzer. The input is voice data, which the emotion analysis engine analyzes to determine the user's emotional state. Specifically, emotions such as vigilance and anxiety are extracted from the voice data. Data processing involves emotion analysis, and the emotion analysis results are generated as output. These results are then sent to the server.

[1719] Step 4:

[1720] Enhanced alert notifications

[1721] The server evaluates the likelihood of fraud by combining the results of text analysis and sentiment analysis. The inputs are the results of text analysis and sentiment analysis, and the server uses these to make a comprehensive judgment on the likelihood of fraud. Data calculation involves integrating the results of the two analyses to calculate the probability of fraud occurring, and the output is an alert notification message. The server then sends this alert notification to the configured contacts.

[1722] Step 5:

[1723] Call blocking

[1724] If the server determines that there is a high possibility of fraud, it sends a call-hanging instruction to the terminal. The terminal receives this instruction and automatically ends the current call. The input is the "call-hanging" instruction from the server, and the output is the call termination process. Specifically, the terminal executes the "call-hanging" command to hang up the call, preventing the user from continuing the conversation with the fraudster.

[1725] Step 6:

[1726] ATM warning message

[1727] If there is a high possibility of fraud, the server sends a warning message to the ATM network that the victim may use. The input is the possibility of fraud and related information, and the server generates a warning message based on this. Data processing involves generating a warning message, and as output, a warning message is generated that is sent to the ATM network. The ATM receives this message, displays a warning to the user, and if certain conditions are met (e.g., withdrawing a large amount), additional restrictions are triggered.

[1728] (Application example 2)

[1729] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1730] While conventional call monitoring systems can detect signs of fraud, they have the problem of being unable to analyze the user's emotions and take appropriate action based on the results. Even if signs of fraud are detected, there is a risk that the damage will increase if the user continues the call. Furthermore, even if a fraudulent call is deemed highly likely, the system lacks the functionality to immediately provide appropriate warnings and countermeasures. Therefore, a multi-stage response that takes user emotions into account is needed.

[1731] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice data during a call in real time and converting the voice into text format, means for analyzing the converted text data and detecting patterns of special fraud, means for sending an alert notification to a set contact if there is a high possibility of fraud, means for cutting off the call if it is determined that there is a high possibility of fraud, means for sending a warning message to the relevant transaction device, means for analyzing the caller's emotions from the collected voice data, and means for strengthening the content of the alert notification based on the emotion analysis results. This enables multi-stage and rapid response.

[1732] "During a call, audio data" refers to audio information collected through the terminal's microphone while a call is in progress.

[1733] "Text format" refers to a format in which audio data is analyzed and converted into text data.

[1734] "Patterns of special fraud" refers to a collection of text data that includes keywords, phrases, and speaking characteristics specific to fraudulent acts.

[1735] "Configured Contacts" are emergency contacts designated by the user in advance, typically trusted family and friends.

[1736] "Alert Notification" means a warning message sent to a Contact when potential fraud is detected.

[1737] "Call blocking" is an operation that forcibly ends a current call if it is determined that there is a high possibility of fraud.

[1738] "Transaction device" refers to a device that conducts financial transactions, such as an ATM, and includes devices that have the function of receiving warning messages.

[1739] A "warning message" is a notification intended to alert a user or transaction device to suspected fraud.

[1740] "Means for analyzing the caller's emotions from collected voice data" refers to technology that uses voice data to analyze the caller's emotional state (such as vigilance or anxiety).

[1741] "Means to enhance the content of alert notifications based on the results of sentiment analysis" refers to an interface that enhances the importance and urgency of alert notifications based on sentiment analysis, encouraging appropriate action.

[1742] This invention is a system that collects voice data during calls in real time, detects signs of special fraud, and analyzes the user's emotions to prevent fraud damage before it occurs. The specific processing flow and details of the program for this system are explained below.

[1743] 1. Collecting and converting audio data into text

[1744] Device:

[1745] When a call is initiated, the device immediately collects voice data during the call. The voice data is captured through the device's microphone and converted into digital form. This collected voice data is immediately sent to a speech recognition engine (e.g., Google Speech-to-Text API) using highly accurate voice recognition technology to convert the voice into corresponding text data.

[1746] 2. Text Data Analysis

[1747] server:

[1748] The converted text data is sent to a server, which analyzes it using natural language processing (NLP) techniques (e.g., spaCy or NLTK). This analysis process includes algorithms that detect keywords and phrases characteristic of fraud. Based on the analysis results, it is determined whether a fraud pattern exists.

[1749] 3. Emotion analysis

[1750] Device:

[1751] In parallel, the voice data during the call is also sent to an emotion engine (for example, IBM Watson Tone Analyzer). The emotion engine recognizes emotions from the user's voice. If emotions such as alertness or anxiety are detected, that information is also sent to the server.

[1752] 4. Enhanced alert notifications

[1753] server:

[1754] If the analysis determines that there is a high possibility of fraud, the results of the sentiment analysis are also taken into account. If the user's sentiment indicates vigilance or anxiety, the alert notification will be enhanced and a message urging immediate action will be generated. The server will then send the alert notification to the configured contacts (usually the victim's family or trusted individuals). The notification can be sent using the Twilio API.

[1755] 5. Call blocking

[1756] Device:

[1757] If a fraudulent call is deemed likely, the device will automatically terminate the current call based on instructions from the server, preventing the victim from further interaction with the fraudster.

[1758] 6. ATM warning messages

[1759] server:

[1760] If fraud is likely, the server will send a warning message to transaction devices that the victim may use (e.g., ATM networks), stating that a fraud is suspected and urging them to be careful.

[1761] Trading Device:

[1762] Upon receiving the warning message, the transaction device will display a warning to the user, and if certain conditions are met (e.g., large withdrawals), additional restrictions may be imposed, including withdrawal limits and additional identity verification procedures.

[1763] Specific examples

[1764] Example 1: Victim receives a call from a scammer

[1765] 1. Device: The victim's mobile phone receives a call from a scammer. The device collects the audio during the call and converts it into text in real time.

[1766] 2. Server: The server analyzes the text data and detects keywords that indicate fraud, such as "bank account," "transfer," and "family emergency."

[1767] 3. Device: At the same time, the emotion engine analyzes the user's emotions from the voice data during the call. If alertness or anxiety is detected, that information is also sent to the server.

[1768] 4. Server: Determines the likelihood of fraud and takes into account sentiment analysis results to create an enhanced alert, which includes information that a suspected fraudulent call has been detected and instructions to take immediate action.

[1769] 5. Device: Call blocking is implemented, preventing the victim from continuing the call with the scammer.

[1770] 6. Server: Sends a warning message to transaction devices that the victim may use, indicating that there is a high possibility of fraud.

[1771] 7. Trading Device: The trading device displays warning messages, notifies users of potential fraud, and triggers additional restrictions based on certain conditions.

[1772] Example prompt sentence:

[1773] "This call shows signs of fraud. It contains the following phrases: bank account, large withdrawal, family emergency. The scam was also confirmed through sentiment analysis. End the call immediately and send an alert notification to a trusted contact."

[1774] In this way, the system of the present invention detects fraudulent behavior in real time during a call and takes multi-stage measures by combining it with emotion analysis, thereby making it possible to prevent fraud damage before it occurs.

[1775] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1776] Step 1:

[1777] Device: When a call is initiated, the device immediately collects audio data during the call. Audio data is captured through the device's microphone and converted into a digital format. This collected audio data is converted into text data in real time using the Google Speech-to-Text API.

[1778] Input: Audio data

[1779] Output: Text data

[1780] What it does: The microphone captures voice data, which is then sent to a speech recognition engine, which converts the speech into text and returns it to the device.

[1781] Step 2:

[1782] Terminal: The converted text data is sent to the server.

[1783] Input: Text data

[1784] Output: Text data sent to the server

[1785] Specific operation: The terminal sends the converted text data to the server via the network.

[1786] Step 3:

[1787] Server: The server analyzes the text data using natural language processing (NLP) techniques (e.g., spaCy or NLTK). The analysis process includes algorithms that detect keywords and phrases characteristic of fraud.

[1788] Input: Text data

[1789] Output: High probability of fraud detection results

[1790] What it does: It feeds text data into an NLP engine to check for the presence of fraud-related keywords, and if a fraud pattern is detected, it flags it accordingly.

[1791] Step 4:

[1792] Terminal: In parallel, the voice data during the call is also sent to an emotion engine (e.g., IBM Watson Tone Analyzer), which recognizes emotions from the user's voice.

[1793] Input: Audio data

[1794] Output: Emotion analysis results

[1795] What it does: It sends voice data to an emotion analysis engine, which analyzes the tone and patterns of the voice to identify the emotional state. If it detects emotions such as alertness or anxiety, it sends the results to a server.

[1796] Step 5:

[1797] Server: If the analysis determines that fraud is likely, the sentiment analysis is also taken into account. An enhanced alert notification is generated and sent to configured contacts.

[1798] Input: Fraud pattern detection results and sentiment analysis results

[1799] Output: Enhanced alert notifications

[1800] What it does: Based on the results of sentiment analysis, it generates prompt text to adjust the content and urgency of fraud notifications, and uses the Twilio API to send alert notifications to configured contacts.

[1801] Step 6:

[1802] Device: If a fraudulent call is deemed likely, the current call will be automatically terminated based on instructions from the server.

[1803] Input: Instructions from the server

[1804] Output: End of call

[1805] Specific operation: Upon receiving a call termination command from the server, the device forcibly terminates the call.

[1806] Step 7:

[1807] Server: If fraud is suspected, it sends a warning message to the victim's potential transaction device. This message notifies the victim of the suspected fraud.

[1808] Input: Fraud pattern detection results

[1809] Output: Warning message to trading device

[1810] What it does: If fraud is suspected, a warning message is sent via a custom API to a transaction device (e.g., an ATM), which displays the message and, if necessary, initiates additional restrictive measures.

[1811] In this way, the system of the present invention performs the necessary data processing and calculation at each step, making it possible to prevent fraud damage through multi-stage responses.

[1812] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1813] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1814] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1815] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1816] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1817] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1818] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1819] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1820] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1821] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1822] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1823] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1824] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1825] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1826] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1827] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1828] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1829] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1830] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1831] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1832] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1833] The following is further disclosed regarding the above embodiment.

[1834] (Claim 1)

[1835] A means for collecting voice data during a call in real time and converting the voice data into text format;

[1836] A means for analyzing the converted text data and detecting patterns of special fraud;

[1837] A means to send alert notifications to configured contacts in case of potential fraud;

[1838] measures to block calls if they are deemed likely to be fraudulent;

[1839] means for sending a warning message to the affected ATM;

[1840] A system including:

[1841] (Claim 2)

[1842] 2. The system according to claim 1, further comprising means for analyzing voice data during a call using natural language processing.

[1843] (Claim 3)

[1844] 10. The system of claim 1, wherein the ATM receiving the warning message has means for displaying the warning and for invoking additional restrictions according to specified conditions.

[1845] "Example 1"

[1846] (Claim 1)

[1847] A means for collecting voice data during a call in real time and converting the voice data into text format;

[1848] A means for analyzing the converted text data using natural language processing technology to detect patterns of special fraud;

[1849] A means to send alert notifications to configured contacts in case of potential fraud;

[1850] measures to block calls if they are deemed likely to be fraudulent;

[1851] means for sending a warning message to the affected ATM;

[1852] a means for the ATM to receive the warning message, display a warning, and invoke additional restrictions in accordance with certain conditions;

[1853] A system including:

[1854] (Claim 2)

[1855] 2. The system according to claim 1, further comprising means for analyzing voice data during a call using natural language processing techniques.

[1856] (Claim 3)

[1857] 10. The system of claim 1, further comprising means for generating prompt sentences using a generative model to assess likelihood of fraud.

[1858] "Application Example 1"

[1859] (Claim 1)

[1860] A means for collecting voice data during a call in real time and converting the voice data into text format;

[1861] A means for analyzing the converted text data and detecting patterns of special fraud;

[1862] A means to send alert notifications to configured contacts in case of potential fraud;

[1863] measures to block calls if they are deemed likely to be fraudulent;

[1864] means for sending a warning message to the affected ATM;

[1865] a means for displaying a warning message on the user's mobile device;

[1866] a means for analyzing text data using a generative AI model;

[1867] A system including:

[1868] (Claim 2)

[1869] 2. The system according to claim 1, further comprising means for analyzing voice data during a call using natural language processing.

[1870] (Claim 3)

[1871] 10. The system of claim 1, wherein the ATM receiving the warning message has means for displaying the warning and for invoking additional restrictions according to specified conditions.

[1872] "Example 2: Combining Emotion Engines"

[1873] (Claim 1)

[1874] A means for collecting voice data during a call in real time and converting the voice data into text format;

[1875] A means for analyzing the converted text data and detecting patterns of special fraud;

[1876] A means of analyzing emotions in real time from collected voice data,

[1877] A means to send alert notifications to configured contacts in case of potential fraud;

[1878] measures to block calls if they are deemed likely to be fraudulent;

[1879] means for transmitting a warning message to the applicable automated teller machine;

[1880] A system including:

[1881] (Claim 2)

[1882] 10. The system of claim 1, further comprising means for analyzing voice data during a call using natural language processing to detect fraud patterns.

[1883] (Claim 3)

[1884] 10. The system of claim 1, wherein an automated teller machine receiving the warning message has means for displaying a warning and for invoking additional restrictions according to specified conditions.

[1885] "Application example 2 when combining emotion engines"

[1886] (Claim 1)

[1887] A means for collecting voice data during a call in real time and converting the voice data into text format;

[1888] A means for analyzing the converted text data and detecting patterns of special fraud;

[1889] A means to send alert notifications to configured contacts in case of potential fraud;

[1890] measures to block calls if they are deemed likely to be fraudulent;

[1891] means for transmitting a warning message to the transaction device;

[1892] A means for analyzing the emotion of a caller from the collected voice data;

[1893] A means to enhance the content of alert notifications based on the results of sentiment analysis;

[1894] A system including:

[1895] (Claim 2)

[1896] 2. The system according to claim 1, further comprising means for analyzing voice data during a call using natural language processing.

[1897] (Claim 3)

[1898] 10. The system of claim 1, wherein the trading device receiving the warning message has means for displaying the warning and for invoking additional restrictions according to specified conditions. [Explanation of symbols]

[1899] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for collecting voice data during a call in real time and converting the voice data into text format; A means for analyzing the converted text data and detecting patterns of special fraud; A means to send alert notifications to configured contacts in case of potential fraud; measures to block calls if they are deemed likely to be fraudulent; means for sending a warning message to the affected ATM; A system including:

2. 2. The system according to claim 1, further comprising means for analyzing voice data during a call by natural language processing.

3. 2. The system of claim 1, wherein the ATM receiving the warning message has means for displaying a warning and for imposing additional restrictions according to specified conditions.

Images