System

Abstract

A system is provided.SOLUTION: A system comprising: means for detecting a start of a call; means for streaming audio of the call to a server in real-time; means for converting the audio to text; means for analyzing the text to determine a likelihood of fraud; means for recording call content if the likelihood of fraud is high; and means for sending a notification if there is a likelihood of fraud.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The number of victims of special telephone fraud, particularly among the elderly, continues to grow, and the resulting losses are substantial. Conventional countermeasures have made it difficult to respond quickly to increasingly sophisticated fraudulent methods, making it difficult to prevent damage. In response to this, there is a need for a system that analyzes call content in real time, detects potentially fraudulent calls early, and prompts appropriate action to prevent damage before it occurs. Therefore, the present invention aims to provide a system that analyzes call content in real time and can respond quickly if it determines that a call may be fraudulent. [Means for solving the problem]

[0005] The present invention provides a system including a means for detecting the start of a call, a means for streaming the call audio to a server in real time, a means for converting the audio into text, a means for analyzing the text and determining the possibility of fraud, a means for recording the call content if there is a high possibility of fraud, and a means for sending a notification if there is a possibility of fraud. This enables the call content to be analyzed in real time, and countermeasures to be taken quickly if there is a possibility of fraud. Furthermore, by using a generative AI model as a means for determining the possibility of fraud, specific keywords and phrases can be detected with high accuracy, and the possibility of fraud can be accurately determined. Furthermore, by having the user or their family receive the notification and review the content of the fraudulent call and take action as necessary, fraud damage can be prevented before it occurs.

[0006] A "means for detecting the start of a call" is a device or software that has the function of automatically recognizing when a user starts a call on the telephone.

[0007] "Means for streaming the voice of a call to a server in real time" refers to a device or software that has the function of transmitting voice data during a call to a server in real time.

[0008] A "means for converting speech to text" is a speech recognition system or software that has the function of analyzing speech data and converting it into corresponding text data.

[0009] "Means for analyzing text to determine likelihood of fraud" refers to a generative AI model or algorithm that has the ability to analyze the converted text data and assess whether it is likely to be fraudulent.

[0010] "Means for recording telephone conversations when fraud is likely" refers to devices or software that automatically record telephone conversations when fraud is deemed to be likely and store the data.

[0011] "Means for sending a notification in the event of possible fraud" refers to a device or software that has the function of sending a warning to the user or a designated notification recipient when it is determined that there is a possibility of fraud. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0013] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0016] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0017] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0018] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0020] [First embodiment]

[0021] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0022] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0025] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0028] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0032] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0033] This invention is a system that analyzes call content in real time and responds quickly to potential fraud. The system monitors call content when a user initiates a call and uses generative AI to detect specific keywords and phrases to determine the possibility of fraud. If fraud is likely, the system records the call and sends a warning to designated notification recipients.

[0034] System Configuration

[0035] 1. Call detection

[0036] The server detects when a user initiates a call on the telephone by receiving a call initiation signal from the user terminal.

[0037] 2. Streaming of call content

[0038] The device streams the call audio to the server in real time. The device has a microphone that converts the audio into digital data and transmits the data to the server.

[0039] 3. Voice Recognition

[0040] The server passes the received voice data to a speech recognition module that converts the voice data to text, which uses a pre-trained model to convert speech to text with high accuracy.

[0041] 4. Text Analysis

[0042] Generative AI analyzes the converted call content to detect specific fraud patterns and keywords, using algorithms to assess the likelihood of fraud.

[0043] 5. Determining the likelihood of fraud

[0044] The generation AI determines whether there is a possibility of fraud based on the analysis results, and if there is a high possibility of fraud, it returns the result to the server.

[0045] 6. Recording and saving of call content

[0046] If the generation AI determines that there is a high possibility of fraud, the server will record the call and store it in secure data storage.

[0047] 7. Sending Notifications

[0048] If the server detects a potentially fraudulent call, it will send a warning to the user and their designated contacts via push notification, email, SMS, or other methods.

[0049] A natural language description of the program's processing

[0050] Call detection

[0051] When a user starts a call, the server detects the call start event and prepares for real-time analysis. For example, when an elderly person starts a call on their home landline phone, the server automatically detects the call.

[0052] Streaming of call content

[0053] The terminal transmits the voice during the call to the server in real time. The actual voice is converted into digital data and sent as packets to the server.

[0054] Voice Recognition

[0055] The server analyzes the received voice data and converts it into text using a speech recognition module, which is then used in the next analysis step.

[0056] Text analytics

[0057] The generation AI analyzes the text data received from the voice recognition module and detects specific keywords and phrases to determine whether the data is fraudulent. For example, if the data contains phrases such as "It's me" or "I need money urgently," the generation AI will detect this.

[0058] Determining the likelihood of fraud

[0059] The generative AI evaluates the likelihood of fraud based on the analysis results and returns the score to the server, flagging cases where fraud is highly likely.

[0060] Call recording and storage

[0061] The server records calls that are deemed likely to be fraudulent and stores the data in secure storage. For example, recording begins when a phrase such as "I want to transfer money" is detected.

[0062] Sending notifications

[0063] The server notifies the user and the designated contacts that a potentially fraudulent call has been detected. The notification may be sent as a warning message such as, "Your mother's call may be fraudulent."

[0064] In this way, the present invention aims to prevent fraud by analyzing the contents of calls in real time, and if there is a possibility of fraud, quickly recording the contents of the call and making appropriate notifications, thereby preventing fraud damage from occurring.

[0065] The processing flow will be explained below.

[0066] Step 1:

[0067] The server waits for and receives a call start signal from the terminal to detect when the user has started a call on the phone, and when the server receives this signal, it starts monitoring the call in real time.

[0068] Step 2:

[0069] The device streams the audio to the server in real time as soon as the user starts a call. The device has a microphone that converts the audio into digital data and sends it to the server.

[0070] Step 3:

[0071] The server passes the received voice data to a speech recognition module, which analyzes the voice data and converts it into corresponding text data in real time, ready for the subsequent analysis step.

[0072] Step 4:

[0073] The Generative AI receives the text data sent from the speech recognition module and begins analyzing it to detect specific fraud patterns and keywords. Using pre-trained models, the Generative AI scores the likelihood of fraud.

[0074] Step 5:

[0075] The generation AI determines whether there is a possibility of fraud based on the analysis results. The generation AI calculates the probability of fraud for each keyword or phrase and returns the result to the server as a score. If there is a high possibility of fraud, it issues instructions to the server.

[0076] Step 6:

[0077] If the server determines that a call is likely to be fraudulent, it automatically records the call audio and stores the recorded audio in secure data storage for later review.

[0078] Step 7:

[0079] When a potentially fraudulent call is detected, the server sends a warning to the user and designated contacts (family and relatives) via push notification, email, SMS, etc.

[0080] Step 8:

[0081] The user and the designated contacts will receive a warning from the server. If the user or their family members determine that the information is likely fraudulent, they can take appropriate action, such as further investigating the matter or reporting it to the police.

[0082] For example, when an elderly person starts a call on their home landline, the server automatically detects the call and analyzes the content in real time. If a phrase such as "I need money urgently" is detected during the call, the server records the call and simultaneously sends a warning to the family. By receiving this warning, the family can prevent fraud before it occurs.

[0083] Example 1

[0084] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0085] In recent years, telephone fraud has been on the rise, with many victims reported, particularly among the elderly. To prevent such fraud, a system is needed that can monitor call content in real time and quickly detect possible fraud. However, current systems have difficulty analyzing call content in real time and accurately determining the possibility of fraud. Therefore, there is a need for a method that can accurately detect possible fraud during a call and respond quickly.

[0086] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0087] In this invention, the server includes means for detecting the start of a call, means for streaming the call audio to the server in real time, means for converting the audio into text, means for analyzing the text using a model for detecting specific keywords and phrases to determine the possibility of fraud, means for recording the content of the call and saving it in secure data storage if there is a high possibility of fraud, means for sending a notification if there is a possibility of fraud, and means for receiving the notification and allowing the user to confirm the content of the fraudulent call. This makes it possible to detect the possibility of fraud during a call with high accuracy in real time and take necessary measures quickly.

[0088] A "means for detecting the start of a call" is a device or system for detecting the moment when a user starts a telephone call.

[0089] "Means for streaming telephone call audio to a server in real time" refers to a device or system for converting telephone call audio into digital data in real time and transmitting it to a server via the Internet.

[0090] A "means for converting speech to text" is a device or system that uses speech recognition technology to analyze received speech data and convert it into text data.

[0091] A "model for detecting specific keywords and phrases" is a trained algorithm used to analyze text data during calls and detect specific fraud-related keywords and phrases.

[0092] The "means for determining the likelihood of fraud" is a device or system that evaluates the likelihood of fraud based on the analyzed text data and provides the results.

[0093] A "call recording device" is a device or system that records calls that are determined to be likely fraudulent and stores the data.

[0094] "Secure Data Storage" means an encrypted data storage device or system for securely storing recordings.

[0095] A "means for sending a notification" is a device or system for sending an alert to designated notification contacts when a potentially fraudulent call is detected.

[0096] The "means for receiving the notification and allowing the user to check the content of the fraudulent call" refers to a device or system that receives the sent notification and allows the user to check the content of the notification.

[0097] This invention is a system that analyzes call content in real time and responds quickly to potential fraud. The system monitors call content as soon as the user initiates a call and uses a generative AI model to detect specific keywords and phrases to determine the likelihood of fraud. If fraud is likely, the system records the call and sends a warning to designated contacts.

[0098] This system is implemented using the following hardware and software:

[0099] Hardware: Servers, user devices (smartphones, landlines, etc.)

[0100] Software: speech recognition modules (e.g., Google Cloud Speech-to-Text), generative AI models (e.g., OpenAI GPT-4)

[0101] Call detection

[0102] When a user starts a call on the phone, the server receives a call start signal from the user terminal and detects the start of the call. For example, when an elderly person starts a call on their home landline phone, the server receives the call start signal emitted from the terminal and confirms that the call has started.

[0103] Streaming of call content

[0104] After a call starts, the device streams the call audio to the server in real time. Specifically, the device's built-in microphone converts the audio into digital data and sends that data to the server via the Internet. For example, a smartphone's microphone picks up the audio, digitizes it, and sends it to the server in real time as packets.

[0105] Voice Recognition

[0106] The server passes the received voice data to a speech recognition module (e.g., Google Cloud Speech-to-Text) and converts the voice into text. For example, if a user says "hello" during a phone conversation, the word will be converted into text as "Konnichiwa."

[0107] Text analytics

[0108] The generation AI analyzes the text data sent from the voice recognition module and detects specific keywords and phrases (e.g., "It's me," "I need money urgently," etc.). For example, if the call text contains the phrase "Please transfer the money immediately," the generation AI will detect this and calculate a score indicating the likelihood of fraud.

[0109] Determining the likelihood of fraud

[0110] The generative AI evaluates the likelihood of fraud based on the analysis results and returns the result to the server. For example, if a specific fraud-related phrase is detected, a high fraud risk score is set and the score is sent to the server.

[0111] Call recording and storage

[0112] The server records calls that are deemed likely to be fraudulent and stores the data in secure data storage. For example, recording begins the moment the user responds to the fraudster by saying, "I'll transfer the money." The recording data is then securely stored in encrypted storage.

[0113] Sending notifications

[0114] The server notifies the user and designated contacts that a potentially fraudulent call has been detected. Notifications can be sent in the form of push notifications, emails, SMS, etc. For example, a warning message stating "Your mother's call may be fraudulent" could be sent to the user's family.

[0115] As described above, the present invention aims to prevent fraud by analyzing the contents of calls in real time, and if there is a possibility of fraud, quickly recording the contents of the call and making appropriate notifications, thereby preventing fraud damage from occurring.

[0116] Prompt Sentence Examples

[0117] Examples of prompts to input to a generative AI model include:

[0118] Please explain in natural language the programming steps of the generative AI that analyzes user conversations in real time and detects specific keywords and phrases. Please also provide specific examples.

[0119] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0120] System program processing flow

[0121] Step 1: Call detection

[0122] Input: Signal to start a call

[0123] Processing: The server receives a call initiation signal from the user terminal to detect that the user is about to initiate a call.

[0124] Specific operation: For example, when an elderly person starts a call using their home landline, the device detects this and sends a signal to the server, which then receives the signal and confirms that the call has started.

[0125] Output: Call start confirmation information

[0126] Step 2: Stream the call

[0127] Input: Call audio data

[0128] Processing: The device streams the call audio to the server in real time, converts the call audio into digital data, and sends the data over the internet to the server.

[0129] How it works: For example, a smartphone's microphone picks up the audio of a call and converts it into digital data, which is then split into packets and streamed to a server.

[0130] Output: Digitized voice data

[0131] Step 3: Voice Recognition

[0132] Input: Digitized voice data

[0133] Processing: The server passes the received voice data to a speech recognition module (e.g., Google Cloud Speech-to-Text) to convert the voice into text.

[0134] Specific operation: For example, if the voice data contains "Hello", it will be converted into text data as "Konnichiwa".

[0135] Output: Text data

[0136] Step 4: Text Analysis

[0137] Input: Text data

[0138] Processing: The generative AI analyzes the text data sent from the speech recognition module to detect specific fraud-related keywords and phrases (e.g., "It's me," "I need money urgently," etc.).

[0139] Specific behavior: For example, if the text data contains the phrase "Please transfer the money immediately," the generative AI will detect this and calculate a score indicating the likelihood of fraud.

[0140] Output: Keyword analysis results and fraud likelihood score

[0141] Step 5: Determine the likelihood of fraud

[0142] Input: Keyword analysis results and fraud likelihood score

[0143] Processing: The generative AI assesses the likelihood of fraud based on the analyzed text data and score.

[0144] Specific behavior: For example, if the fraud likelihood score is set high, the generation AI will return that score to the server.

[0145] Output: Fraud likelihood assessment result

[0146] Step 6: Record and save the call

[0147] Input: Fraud likelihood assessment results

[0148] Processing: The server records calls that are determined to be likely fraudulent and stores the data in secure storage.

[0149] Specific operation: For example, the moment the user responds to the scammer by saying "I'll transfer the money," recording begins, and the recording data is stored in encrypted storage.

[0150] Output: Recorded and securely stored data

[0151] Step 7: Sending notifications

[0152] Input: Fraud likelihood assessment results and audio recording

[0153] Action: The server notifies the user and designated contacts that a potentially fraudulent call has been detected.

[0154] Specific behavior: Notifications can be sent in the form of push notifications, emails, SMS, etc. For example, a warning message stating "Your mother's call may be fraudulent" is sent to the user's family.

[0155] Output: Information message

[0156] Through these steps, the system can analyze call content in real time, quickly detect potential fraud, and take necessary action.

[0157] (Application example 1)

[0158] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0159] In modern society, telephone fraud continues to affect many people. Elderly and other vulnerable groups are particularly susceptible to fraud, easily falling prey to the sophisticated tactics of criminals. Conventional methods have made it difficult to monitor phone conversations in real time, quickly identify potential fraud, and respond accordingly. There is also a need for a system that can quickly notify a third party of potentially fraudulent calls and take appropriate action. Given this background, there is an urgent need to develop a system that can analyze potential fraud in real time during a call and take appropriate action.

[0160] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0161] In this invention, the server includes means for detecting the start of a call, means for streaming the call audio to the server in real time, means for converting the audio to text, means for analyzing the text to determine the possibility of fraud, means for recording the call if fraud is likely, means for sending a notification if fraud is likely, means for including a generative AI model for analyzing the text of the call, and means for sending a notification of suspected fraud to a notification recipient if fraud is likely. This enables fraudulent activity during a call to be detected in real time and appropriate action to be taken promptly.

[0162] The "means for detecting the start of a call" is a device or software that has the function of recognizing and notifying that a call has started.

[0163] "Means for streaming the audio of a call to a server in real time" refers to a device or software that has the function of converting audio data during a call into a digital format in real time and transmitting it to a server.

[0164] A "means for converting speech to text" is a device or software that has the function of analyzing speech data and converting it into corresponding text data.

[0165] A "means for analyzing text to determine likelihood of fraud" is a device or software capable of examining text data to identify fraud-related keywords and phrases and assess the likelihood of fraud.

[0166] "Means for recording telephone conversations when fraud is likely" refers to devices or software that have the function of automatically recording telephone conversations when fraud is deemed to be likely.

[0167] A "means for sending notifications in the event of potential fraud" is a device or software capable of sending a warning message or notification to a specified recipient in the event of potential fraud.

[0168] A "means including a generative AI model for analyzing the text of a call" is a device or software that uses generative AI to convert voice data into text and analyzes that text to detect specific keywords or phrases.

[0169] "Means for sending a notice of suspected fraud to a notification recipient when there is a high possibility of fraud" refers to a device or software that has the function of notifying a pre-determined notification recipient of suspected fraud when it is determined that there is a high possibility of fraud.

[0170] The present invention provides a system that analyzes the contents of calls in real time and responds quickly when there is a possibility of fraud. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail the preferred embodiments of the present invention.

[0171] System Configuration

[0172] The system is realized using the user's smartphone, a server, and a generative AI model.

[0173] Hardware and software used

[0174] Hardware: Smartphones, servers

[0175] Software: Android / iOS SDK, Google Cloud Speech-to-Text API, Fraud Detection AI Model

[0176] System Operation

[0177] 1. Call detection:

[0178] The user's smartphone detects the start of a call, which is notified using the phone application's integration API.

[0179] 2. Streaming Call Content:

[0180] During a call, the smartphone streams the audio in real time to a server, using the smartphone's built-in microphone to convert the audio data into a digital format and send it to the server.

[0181] 3. Speech Recognition:

[0182] The server converts the received voice data into text using the Google Cloud Speech-to-Text API.

[0183] 4. Text Analysis:

[0184] The server uses a generative AI model (e.g., based on BERT or GPT-4) to analyze the text data and detect keywords and phrases related to fraud.

[0185] 5. Determining the likelihood of fraud:

[0186] The server scores the likelihood of fraud based on the keywords and phrases detected.

[0187] 6. Recording and storage of calls:

[0188] If the server determines that there is a high possibility of fraud, it will automatically record the call and store it in secure storage (e.g., Amazon S3).

[0189] 7. Sending Notifications:

[0190] If there is a high possibility of fraud, the server will send a warning message or a notification to the user and pre-specified notification destinations using a notification API such as Twilio.

[0191] Specific examples

[0192] For example, if an elderly parent receives a call from a scammer, here's how it works:

[0193] 1. An elderly parent's smartphone detects the start of a call.

[0194] 2. The call content is sent to the server in real time.

[0195] 3. The server converts the speech into text through speech recognition.

[0196] 4. The generative AI model detects fraudulent keywords such as "It's me" and "I need money."

[0197] 5. If a call is deemed likely to be fraudulent, the call will be recorded and a warning will be sent to the user and their family.

[0198] 6. A warning message will be sent, for example, a notification saying "Your mother's call may be fraudulent."

[0199] Prompt Sentence Examples

[0200] "I want to create an app to protect against fraud. The app will analyze phone calls in real time, detect specific fraudulent keywords and phrases, and issue a warning. The prompt might look like this:

[0201] Please analyze this call to see if it contains any scam phrases, and if so, please let us know what they are."

[0202] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0203] Step 1:

[0204] The smartphone detects the start of a call

[0205] Specific operation: When a user starts a call using the phone app, a call start event occurs, and the start of the call is notified to the server via the smartphone's collaboration API. This event is sent to the server as call log information.

[0206] Input: Signal to start a call

[0207] Output: Call start notification and call log information

[0208] Step 2:

[0209] The smartphone streams the call content to the server in real time.

[0210] What it does: It uses the smartphone's microphone to capture the audio during a call and streams it to a server as digital audio data, which is sent in packets.

[0211] Input: Call audio

[0212] Output: Streaming digital audio data

[0213] Step 3:

[0214] The server passes the voice data to the speech recognition module and converts it into text.

[0215] What it does: It passes the received digital audio data to the Google Cloud Speech-to-Text API, which converts the audio into text, which is then used in the next analysis step.

[0216] Input: Digital audio data

[0217] Output: Text data

[0218] Step 4:

[0219] The server uses a generative AI model to analyze the text data and detect fraudulent keywords and phrases.

[0220] How it works: Using a generative AI model (such as BERT or GPT-4) to detect specific keywords or phrases in text data, such as "It's me" or "I need money urgently."

[0221] Input: Text data

[0222] Output: Fraudulent keyword and phrase detection results

[0223] Step 5:

[0224] The server scores the likelihood of fraud based on detected keywords and phrases.

[0225] What it does: Based on the keywords and phrases detected, it applies an algorithm to assess the likelihood of fraud and calculates a fraud risk score. It then determines the risk of the call.

[0226] Input: Fraudulent keyword or phrase detection results

[0227] Output: Fraud risk score

[0228] Step 6:

[0229] The server records potentially fraudulent calls and stores them in secure storage.

[0230] What it does: If the fraud risk score is determined to be high, the system will begin recording the call and store the recording in secure cloud storage such as AWS S3. After recording is complete, the data is encrypted and stored.

[0231] Input: Fraud Risk Score

[0232] Output: Save call recording data

[0233] Step 7:

[0234] Sends a warning notification if the server is at high fraud risk

[0235] Specific behavior: If the fraud risk score is high, a warning message is sent to the user and pre-designated notification recipients. Notifications such as "Possible fraud" are sent via push notification, email, or SMS using notification APIs such as Twilio.

[0236] Input: Fraud Risk Score

[0237] Output: Sending a warning message

[0238] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0239] The present invention combines a system that analyzes call content in real time and responds quickly to potential fraudulent activity with an emotion engine that recognizes user emotions. The system monitors call content from the moment the user initiates a call and analyzes specific keywords, phrases, and even voice characteristics to more accurately determine the likelihood of fraud. If a fraudulent activity is suspected, the system records the call and sends a warning to designated notification recipients.

[0240] System Configuration

[0241] 1. Call detection

[0242] The server waits for and receives a call start signal from the terminal to detect when the user has started a call on the phone, and when the server receives this signal, it starts call monitoring in real time.

[0243] 2. Streaming of call content

[0244] The terminal streams the voice to the server in real time when the user starts a call. The terminal has the function of converting the voice during the call into digital data and sending this data to the server.

[0245] 3. Voice Recognition

[0246] The server passes the received voice data to a speech recognition module, which converts the voice into text using a pre-trained model.

[0247] 4. Text Analysis

[0248] The generative AI analyzes the text data sent from the voice recognition module to detect specific fraud patterns and keywords, while also analyzing voice characteristics using an emotion engine to recognize the user's emotions.

[0249] 5. Determining the likelihood of fraud

[0250] The generative AI and emotion engine combine the results of their analysis to determine the likelihood of fraud. The generative AI calculates the probability of fraud for each keyword or phrase, and then takes into account the results of the emotion engine's analysis of user sentiment to precisely assess the likelihood of fraud.

[0251] 6. Recording and saving of call content

[0252] The server records calls that are deemed likely to be fraudulent and stores the data in secure storage, which can then be used for later review or as evidence.

[0253] 7. Sending Notifications

[0254] When a potentially fraudulent call is detected, the server sends a warning to the user and designated contacts (family and relatives). Notification methods can be selected from push notifications, emails, SMS, etc.

[0255] A natural language description of the program's processing

[0256] Call detection

[0257] When a user starts a call, the server detects the call start event and prepares for real-time analysis. For example, when an elderly person starts a call on their home landline phone, the server automatically detects the call.

[0258] Streaming of call content

[0259] The device transmits the call audio to the server in real time. The device is equipped with a microphone that converts the audio into digital data and sends it to the server as packets.

[0260] Voice Recognition

[0261] The server passes the received voice data to a speech recognition module, which converts the voice data into text data that is used in further analysis steps.

[0262] Text and Sentiment Analysis

[0263] The generative AI analyzes the text data sent from the voice recognition module to detect specific keywords and phrases. At the same time, the emotion engine analyzes the voice characteristics to recognize the user's emotions. For example, the emotion engine analyzes the tone, speed, and volume of the user's voice to determine stress or urgency.

[0264] Determining the likelihood of fraud

[0265] Generative AI combines the results of analytics and sentiment analysis to assess the likelihood of fraud. This process involves scoring the fraud risk for each keyword or phrase, and then taking into account the results of the sentiment engine to make a final decision.

[0266] Call recording and storage

[0267] The server records any audio data that the AI ​​determines to be fraudulent and stores it in secure storage. For example, recording begins when a phrase such as "transfer your money now" is detected.

[0268] Sending notifications

[0269] The server will notify the user and the designated contacts that a potentially fraudulent call has been detected. The notification will be sent as a warning message saying, "Your mother's call may be fraudulent."

[0270] In this way, the present invention provides a system that analyzes call content in real time and combines it with an emotion engine to more accurately determine the possibility of fraud and respond quickly. Specifically, the system automatically starts working the moment the user starts a call, and if a possible fraud is detected, a notification is sent immediately, preventing fraud damage before it occurs.

[0271] The processing flow will be explained below.

[0272] Step 1:

[0273] The server detects when a user starts a call on the phone. The server receives a call start signal from the user's device and prepares for real-time call monitoring. Specifically, the server receives a signal from the device as soon as the call starts and starts the analysis module.

[0274] Step 2:

[0275] When a user initiates a call, the device streams the call audio to the server in real time. The device digitizes the audio picked up through the microphone and sends the data to the server. For example, audio data from a landline phone or smartphone is continuously sent to the server.

[0276] Step 3:

[0277] The server passes the received voice data to a speech recognition module, which converts it into text data. The speech recognition module uses a pre-trained model to analyze the voice and convert it into corresponding text, which is used in the next analysis step.

[0278] Step 4:

[0279] The generation AI receives the text data sent from the speech recognition module and begins analyzing it to detect specific fraud patterns and keywords. At the same time, the emotion engine analyzes this voice data to recognize the user's emotions. For example, the generation AI detects fraud-specific phrases such as "I need money urgently," and the emotion engine analyzes the user's tone and speed of voice to indicate feelings of tension or impatience.

[0280] Step 5:

[0281] The generative AI and emotion engine combine the analysis results to determine the likelihood of fraud. First, the generative AI calculates a fraud score for the detected keywords and phrases, and then the emotion engine makes a final assessment based on the user's emotion results identified by the generative AI. If it determines that there is a high likelihood of fraud, it returns the result to the server.

[0282] Step 6:

[0283] The server then issues instructions to record calls deemed likely to be fraudulent and store the data in secure storage, where it can later be used for verification or evidence. This step is triggered if specific phrases, such as "I need money now," are detected.

[0284] Step 7:

[0285] If the server detects a potentially fraudulent call, it will send a warning to the user and designated recipients (for example, family and relatives). Notification methods can be selected from push notifications, emails, SMS, etc. For example, a warning message stating, "There is a possibility that your mother's call may be fraudulent" will be sent.

[0286] Step 8:

[0287] The user and the designated contacts receive the warning from the server and check the details. The contacts, such as family members, check the warning message and, if necessary, contact the elderly person again to check the details of the call. At this time, they can also notify the police if necessary and take detailed measures.

[0288] In this way, the present invention analyzes the content of phone calls in real time and combines it with an emotion engine to create a system that can accurately determine the possibility of fraud and respond quickly, thereby preventing fraud damage before it occurs.

[0289] Example 2

[0290] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0291] As fraudulent phone calls increase, there is a problem that vulnerable people, such as the elderly, are particularly susceptible to these attacks. Conventional methods have not established a means to detect fraud during a call and respond quickly. Furthermore, there is a need for a highly accurate analysis method that takes into account the user's emotions.

[0292] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for detecting the start of a call, means for streaming the voice of the call to the central processing unit in real time, means for converting the voice to text, means for analyzing the text to determine the possibility of fraud and the user's emotions, means for recording the contents of the call if there is a high possibility of fraud, and means for sending a notification if there is a possibility of fraud. This makes it possible to accurately determine the possibility of fraud during a call and respond quickly.

[0293] The "means for detecting the start of a call" is a function that automatically recognizes the timing when the user starts a call and notifies the system of the start of the call.

[0294] "Means for streaming the voice of a call to a central processing unit in real time" refers to a function that converts the voice of a call into digital data and instantly transmits that data to a central processing unit.

[0295] The "means for converting voice to text" is a function that analyzes received voice data and converts it into text data.

[0296] The "means for analyzing text to determine the likelihood of fraud and the user's emotions" is a function that uses a generative language model and an emotion analysis device to evaluate the likelihood of fraud and the user's emotions from text data.

[0297] "Means for recording phone calls when there is a high risk of fraud" refers to a function that records the contents of phone calls as digital data when text analysis and sentiment analysis indicate a high risk of fraud.

[0298] "Means for sending notifications in the event of possible fraud" is a function that sends a warning message to the user and designated notification recipients when a risk of fraud is detected.

[0299] A "generative language model" is an algorithm that has been trained in advance on a large amount of language data, and is used to analyze input text data and detect specific patterns and keywords.

[0300] An "emotion analysis device" is a software or hardware device that recognizes a user's emotions from voice data or text data.

[0301] The present invention combines a system that analyzes call content in real time and responds quickly to potential fraud, with an emotion engine that recognizes user emotions. Specific embodiments of this system are described below.

[0302] The system mainly consists of the following elements:

[0303] server

[0304] Terminal

[0305] Voice Recognition Module

[0306] Generation AI

[0307] Emotion Engine

[0308] Secure Storage

[0309] Notification System

[0310] The server detects when a user starts a call by receiving a signal from the terminal. Upon receiving the call start signal, the server starts real-time analysis for call monitoring. When the terminal starts a call, it converts the voice data into digital data and streams it to the server in real time.

[0311] The speech recognition module converts the voice data received by the server into text data. This uses a generative AI model that has been trained on a large amount of data in advance. The generative AI model converts voice to text with high accuracy, removes background noise, and identifies the speaker.

[0312] The generative AI analyzes the text data sent from the speech recognition module to detect specific keywords and phrases. At the same time, the emotion engine analyzes voice characteristics (tone, speed, volume, etc.) to recognize the user's emotions. For example, if the user's voice sounds tense or urgent, the emotion engine will determine the level of stress or urgency.

[0313] Based on the results of this analysis, the generative AI performs a process to assess the likelihood of fraud. It scores the risk of fraud and integrates the results of the emotion engine to make a final judgment. For example, if a phrase such as "Please tell me the bank details" is detected and the tone of voice used is unstable, it will be assessed as having a high likelihood of fraud.

[0314] If the AI ​​generator determines that a call is likely to be fraudulent, the server records the call and stores the data in secure storage, which has access restrictions and can be accessed as needed.

[0315] Furthermore, when a potentially fraudulent call is detected, the server sends a warning to the user and designated contacts. Notification methods include push notification, email, and SMS. For example, a warning message stating, "Your mother's call may be fraudulent" is sent to the contact.

[0316] For example, if an elderly person utters the phrase "Please transfer the money now" during a call, the system will immediately detect this phrase and determine that it is likely a fraud. In this case, the server will record the call, store the data in secure storage, and send an alert to the designated notification recipient.

[0317] Examples of prompts include:

[0318] "Design a fraud detection and notification system when an elderly person utters a phrase that could be fraudulent during a call. The system should include a server that analyzes the call audio in real time, evaluates the possibility of fraud, and records the call and sends a notification if necessary."

[0319] In this way, by analyzing the content of phone calls in real time and combining it with an emotion engine, the present invention can accurately determine the possibility of fraud and respond quickly. The system operates automatically from the moment the user starts a call, and if a possible fraud is detected, a notification is sent immediately, preventing fraud damage before it occurs.

[0320] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0321] Processing Steps

[0322] Step 1:

[0323] The server detects that a user has started a call. It receives a call start signal from the terminal as input and processes this signal as a call start event. This causes the server to start preparing for call monitoring and prepare for real-time voice data analysis. Specifically, when the server receives the call start signal, it starts the analysis module.

[0324] Step 2:

[0325] The terminal converts the call voice into digital data and streams it to the server in real time. As input, the user's voice is collected through a microphone and the voice data is converted into digital data. As output, this digital data is divided into packets and sent to the server. Specifically, the voice conversion module of the terminal generates digital voice data, and the network module sends it to the server.

[0326] Step 3:

[0327] The server passes the received voice data to a voice recognition module, which converts the voice into text data. As input, the digital voice data transmitted in real time is passed to the voice recognition module. As output, this voice data is converted into text data, which is used for the next analysis step. Specifically, the voice recognition module analyzes the voice data using a generative AI model and generates text data.

[0328] Step 4:

[0329] The generation AI analyzes the text data sent from the voice recognition module to detect specific keywords and phrases. At the same time, the emotion engine analyzes the voice characteristics and recognizes the user's emotions. Text data and voice characteristics data are provided as input. The output is an assessment of the likelihood of fraud and the user's emotional state. Specifically, the generation AI checks for the presence of specified keywords and phrases, and the emotion engine analyzes the tone, speed, and volume of the voice.

[0330] Step 5:

[0331] Generative AI integrates the results of analysis and sentiment analysis to assess the likelihood of fraud. The inputs are the detection results of specific keywords and phrases and the analysis results of the sentiment engine. The output is a fraud risk score and a final judgment. Specifically, a risk score is calculated for each keyword or phrase, and the results of sentiment analysis are then added to make the final judgment.

[0332] Step 6:

[0333] If the generation AI determines that there is a high possibility of fraud, the server records the call and stores the data in secure storage. The input is call data with a high fraud risk assessment. The output is the recorded voice data, which is encrypted and stored. Specifically, the call is recorded in real time, and the data is sent to secure storage via an encryption module.

[0334] Step 7:

[0335] The server notifies the user and designated contacts that a potentially fraudulent call has been detected. Event information about the call that was determined to be at high risk of fraud is provided as input. A warning message is sent as output via push notification, email, SMS, etc. Specifically, a notification API is used to send a message to the user or family member stating, "There has been a potentially fraudulent call."

[0336] This is the flow of the program processing for this system. At each step, specific data processing and calculations are performed, allowing for rapid response in cases where there is a high possibility of fraud.

[0337] (Application example 2)

[0338] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0339] In recent years, telephone fraud has been on the rise, with elderly people being particularly vulnerable to its fallout. Because such fraudulent activities use sophisticated persuasive and psychological tactics, it is difficult to ensure sufficient detection accuracy by simply analyzing keywords and phrases in phone call content. Furthermore, even if a possible fraud is detected, it is difficult to prevent damage without a rapid response. Therefore, a system is needed that can accurately determine the possibility of fraud and respond quickly by taking into account the user's emotional state in addition to real-time analysis of phone call content.

[0340] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for detecting the start of a call, means for streaming the call audio to the server in real time, means for converting the audio into text, and means for analyzing the text to determine the possibility of fraud. This enables real-time analysis of the call content. The system also includes means for recording the call content when there is a high possibility of fraud, means for sending a notification when there is a possibility of fraud, and an emotion engine for analyzing the user's emotions in real time, and is equipped with means for precisely evaluating the possibility of fraud by integrating the generative AI model and the emotion engine. This improves the accuracy of determining the possibility of fraud and enables prompt prevention of damage.

[0341] The "means for detecting the start of a call" is a function for automatically detecting the moment when a user starts a call on the telephone.

[0342] "Means for streaming the voice of a call to a server in real time" refers to a function that digitizes the voice data during a call in real time and transmits it to a server via the Internet.

[0343] "Means for converting voice to text" refers to voice recognition technology that analyzes received voice data and converts it into text data.

[0344] "Means for analyzing text to determine the possibility of fraud" refers to a function that analyzes text data converted from speech and detects specific fraudulent keywords and phrases.

[0345] "Means for recording phone calls when there is a high possibility of fraud" refers to a function that records and saves as a digital audio file any phone call that is determined to be highly likely to be fraudulent as a result of analysis.

[0346] "Means for sending notifications in the event of possible fraud" is a function that sends a warning to the user and designated notification recipients when a call with a high risk of fraud is detected.

[0347] The "emotion engine that analyzes user emotions in real time" is a technology that analyzes voice characteristics (e.g., tone, speed, volume) during a call and recognizes the user's emotional state (e.g., stress, urgency) in real time.

[0348] A "generative AI model" is an artificial intelligence model that has been trained in advance with large amounts of data, and is a technology used to detect specific keywords and phrases with high accuracy.

[0349] "Means of precisely assessing the likelihood of fraud by integrating an emotion engine" is a function that integrates the text analysis results of the generative AI model with the emotion analysis results of the emotion engine to comprehensively determine the likelihood of fraud and improve its accuracy.

[0350] The "means for receiving notifications and allowing users to check the content of fraudulent calls" is an interface function that allows users to check the warning notifications they receive and view the details of calls suspected to be fraudulent.

[0351] System program generation

[0352] Hardware and Software Use

[0353] The system for implementing this invention mainly consists of a server and a user's smartphone. The server is equipped with a high-speed processor and a large-capacity SSD, and runs a speech recognition model (e.g., Google Speech-to-Text), a sentiment analysis engine (e.g., IBM Watson Tone Analyzer), and a generative AI model (e.g., OpenAI's GPT-4).

[0354] A smartphone uses the following hardware and software:

[0355] Microphone: Captures call audio.

[0356] Communications module: Streams call audio to the server in real time.

[0357] Internet connection: Allows for high speed transmission of data.

[0358] A natural language description of the program's processing

[0359] Detecting the start of a call

[0360] When a user starts a call on the phone, the smartphone application detects the call start event and notifies the server, which receives the notification and begins preparing for real-time analysis.

[0361] Streaming call audio

[0362] The user's smartphone picks up the call audio with a microphone, converts it into digital data, and sends this data over the Internet to a server, which receives the audio streaming data in real time.

[0363] Speech-to-text

[0364] The server converts the received voice data into text using a speech recognition model (e.g., Google Speech-to-Text), which is then used for subsequent analysis.

[0365] Text and Sentiment Analysis

[0366] A server-based generative AI model (e.g., GPT-4) analyzes the text data obtained from the speech recognition module to detect specific fraudulent keywords and phrases. At the same time, a sentiment analysis engine (e.g., IBM Watson Tone Analyzer) analyzes voice characteristics to recognize the user's emotions in real time. This process is carried out using advanced algorithms.

[0367] Determining the likelihood of fraud

[0368] The server combines the results of the generative AI model and the sentiment analysis engine to precisely assess the likelihood of fraud. It scores the fraud risk by combining specific keywords and the user's emotional state (e.g., high stress). If it determines there is a high likelihood of fraud, the server records the call and stores it in secure storage.

[0369] Warning notice

[0370] The server sends a warning to the user and pre-designated notification recipients (e.g., family members) for calls that are determined to be highly likely to be fraudulent. This notification can be done by push notification, email, SMS, or other means, and will send a message such as, "Signs of fraud have been detected in the content of your call."

[0371] Examples of concrete examples and prompts

[0372] When an elderly person receives a scam call pretending to be their grandchild, the emotion analysis engine detects that the tone and speed of the elderly person's voice warrants attention. The generative AI model picks up phrases such as "transfer money." This causes the server to assess the fraud risk as high, and recording begins. A notification is also sent to the family saying, "There is a possibility that your mother's call is a scam."

[0373] Prompt Sentence Examples

[0374] "Detect potential fraudulent elements in the following phone calls."

[0375] "Assess the user's stress level based on tone of voice and emotional analysis."

[0376] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0377] Step 1:

[0378] Detect the start of a call

[0379] Input: The user initiates a call.

[0380] The server receives a call start event from the smartphone application and detects the start of a call, which allows the server to prepare for real-time analysis.

[0381] Output: Call start information.

[0382] Specific behavior: When a user starts a call on their smartphone, the application catches the event and sends a call start signal to the server.

[0383] Step 2:

[0384] Streaming call audio

[0385] Input: Smartphone call audio.

[0386] The smartphone picks up the call audio through a microphone, converts it into digital data, and streams this data to a server over the internet.

[0387] Output: Real-time audio data.

[0388] How it works: The smartphone's microphone picks up sound, converts the sound signal into digital packets in real time, and sends them to a server. Because this data is sent in real time, it can be analyzed without delay.

[0389] Step 3:

[0390] Speech-to-text

[0391] Input: Real-time audio data.

[0392] The server sends the received voice data to a speech recognition model (e.g., Google Speech-to-Text) and converts the voice into text.

[0393] Output: Text data.

[0394] How it works: A server-based speech recognition model analyzes the audio data and converts it into corresponding text data, which is then used in subsequent analysis steps.

[0395] Step 4:

[0396] Text and Sentiment Analysis

[0397] Input: Text data of speech recognition results.

[0398] A generative AI model (e.g., GPT-4) analyzes text data to detect specific fraudulent keywords and phrases, while an emotion engine (e.g., IBM Watson Tone Analyzer) analyzes voice characteristics to assess user sentiment in real time.

[0399] Output: Deception probability and emotional state assessment results.

[0400] How it works: A generative AI model examines text to identify keywords and phrases related to fraud, while an emotion engine analyzes the tone, rate, and volume of speech to identify emotional states.

[0401] Step 5:

[0402] Determining the likelihood of fraud

[0403] Input: Deception likelihood and emotional state assessment results.

[0404] The server combines the results of the generative AI model and the emotion engine to assess the likelihood of fraud, and if the fraud risk is determined to be high, the call is recorded.

[0405] Output: Fraud risk score and audio recording.

[0406] How it works: The server combines the results of text analysis and sentiment analysis to score the risk of fraud. If the score exceeds a threshold, the call is recorded and stored in a secure location.

[0407] Step 6:

[0408] Warning notice

[0409] Input: Fraud risk score and audio recording.

[0410] The server will send a warning to the user and designated contacts if fraud is likely.

[0411] Output: A warning message.

[0412] What happens: The server uses the notification system to send a warning message to the user and their family members via push notification, email, SMS, etc., such as "Possible fraud detected."

[0413] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0414] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0415] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0416] [Second embodiment]

[0417] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0418] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0419] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0420] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0421] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0422] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0423] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0424] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0425] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0426] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0427] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0428] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0429] This invention is a system that analyzes call content in real time and responds quickly to potential fraud. The system monitors call content when a user initiates a call and uses generative AI to detect specific keywords and phrases to determine the possibility of fraud. If fraud is likely, the system records the call and sends a warning to designated notification recipients.

[0430] System Configuration

[0431] 1. Call detection

[0432] The server detects when a user initiates a call on the telephone by receiving a call initiation signal from the user terminal.

[0433] 2. Streaming of call content

[0434] The device streams the call audio to the server in real time. The device has a microphone that converts the audio into digital data and transmits the data to the server.

[0435] 3. Voice Recognition

[0436] The server passes the received voice data to a speech recognition module that converts the voice data to text, which uses a pre-trained model to convert speech to text with high accuracy.

[0437] 4. Text Analysis

[0438] Generative AI analyzes the converted call content to detect specific fraud patterns and keywords, using algorithms to assess the likelihood of fraud.

[0439] 5. Determining the likelihood of fraud

[0440] The generation AI determines whether there is a possibility of fraud based on the analysis results, and if there is a high possibility of fraud, it returns the result to the server.

[0441] 6. Recording and saving of call content

[0442] If the generation AI determines that there is a high possibility of fraud, the server will record the call and store it in secure data storage.

[0443] 7. Sending Notifications

[0444] If the server detects a potentially fraudulent call, it will send a warning to the user and their designated contacts via push notification, email, SMS, or other methods.

[0445] A natural language description of the program's processing

[0446] Call detection

[0447] When a user starts a call, the server detects the call start event and prepares for real-time analysis. For example, when an elderly person starts a call on their home landline phone, the server automatically detects the call.

[0448] Streaming of call content

[0449] The terminal transmits the voice during the call to the server in real time. The actual voice is converted into digital data and sent as packets to the server.

[0450] Voice Recognition

[0451] The server analyzes the received voice data and converts it into text using a speech recognition module, which is then used in the next analysis step.

[0452] Text analytics

[0453] The generation AI analyzes the text data received from the voice recognition module and detects specific keywords and phrases to determine whether the data is fraudulent. For example, if the data contains phrases such as "It's me" or "I need money urgently," the generation AI will detect this.

[0454] Determining the likelihood of fraud

[0455] The generative AI evaluates the likelihood of fraud based on the analysis results and returns the score to the server, flagging cases where fraud is highly likely.

[0456] Call recording and storage

[0457] The server records calls that are deemed likely to be fraudulent and stores the data in secure storage. For example, recording begins when a phrase such as "I want to transfer money" is detected.

[0458] Sending notifications

[0459] The server notifies the user and the designated contacts that a potentially fraudulent call has been detected. The notification may be sent as a warning message such as, "Your mother's call may be fraudulent."

[0460] In this way, the present invention aims to prevent fraud by analyzing the contents of calls in real time, and if there is a possibility of fraud, quickly recording the contents of the call and making appropriate notifications, thereby preventing fraud damage from occurring.

[0461] The processing flow will be explained below.

[0462] Step 1:

[0463] The server waits for and receives a call start signal from the terminal to detect when the user has started a call on the phone, and when the server receives this signal, it starts monitoring the call in real time.

[0464] Step 2:

[0465] The device streams the audio to the server in real time as soon as the user starts a call. The device has a microphone that converts the audio into digital data and sends it to the server.

[0466] Step 3:

[0467] The server passes the received voice data to a speech recognition module, which analyzes the voice data and converts it into corresponding text data in real time, ready for the subsequent analysis step.

[0468] Step 4:

[0469] The Generative AI receives the text data sent from the speech recognition module and begins analyzing it to detect specific fraud patterns and keywords. Using pre-trained models, the Generative AI scores the likelihood of fraud.

[0470] Step 5:

[0471] The generation AI determines whether there is a possibility of fraud based on the analysis results. The generation AI calculates the probability of fraud for each keyword or phrase and returns the result to the server as a score. If there is a high possibility of fraud, it issues instructions to the server.

[0472] Step 6:

[0473] If the server determines that a call is likely to be fraudulent, it automatically records the call audio and stores the recorded audio in secure data storage for later review.

[0474] Step 7:

[0475] When a potentially fraudulent call is detected, the server sends a warning to the user and designated contacts (family and relatives) via push notification, email, SMS, etc.

[0476] Step 8:

[0477] The user and the designated contacts will receive a warning from the server. If the user or their family members determine that the information is likely fraudulent, they can take appropriate action, such as further investigating the matter or reporting it to the police.

[0478] For example, when an elderly person starts a call on their home landline, the server automatically detects the call and analyzes the content in real time. If a phrase such as "I need money urgently" is detected during the call, the server records the call and simultaneously sends a warning to the family. By receiving this warning, the family can prevent fraud before it occurs.

[0479] Example 1

[0480] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0481] In recent years, telephone fraud has been on the rise, with many victims reported, particularly among the elderly. To prevent such fraud, a system is needed that can monitor call content in real time and quickly detect possible fraud. However, current systems have difficulty analyzing call content in real time and accurately determining the possibility of fraud. Therefore, there is a need for a method that can accurately detect possible fraud during a call and respond quickly.

[0482] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0483] In this invention, the server includes means for detecting the start of a call, means for streaming the call audio to the server in real time, means for converting the audio into text, means for analyzing the text using a model for detecting specific keywords and phrases to determine the possibility of fraud, means for recording the content of the call and saving it in secure data storage if there is a high possibility of fraud, means for sending a notification if there is a possibility of fraud, and means for receiving the notification and allowing the user to confirm the content of the fraudulent call. This makes it possible to detect the possibility of fraud during a call with high accuracy in real time and take necessary measures quickly.

[0484] A "means for detecting the start of a call" is a device or system for detecting the moment when a user starts a telephone call.

[0485] "Means for streaming telephone call audio to a server in real time" refers to a device or system for converting telephone call audio into digital data in real time and transmitting it to a server via the Internet.

[0486] A "means for converting speech to text" is a device or system that uses speech recognition technology to analyze received speech data and convert it into text data.

[0487] A "model for detecting specific keywords and phrases" is a trained algorithm used to analyze text data during calls and detect specific fraud-related keywords and phrases.

[0488] The "means for determining the likelihood of fraud" is a device or system that evaluates the likelihood of fraud based on the analyzed text data and provides the results.

[0489] A "call recording device" is a device or system that records calls that are determined to be likely fraudulent and stores the data.

[0490] "Secure Data Storage" means an encrypted data storage device or system for securely storing recordings.

[0491] A "means for sending a notification" is a device or system for sending an alert to designated notification contacts when a potentially fraudulent call is detected.

[0492] The "means for receiving the notification and allowing the user to check the content of the fraudulent call" refers to a device or system that receives the sent notification and allows the user to check the content of the notification.

[0493] This invention is a system that analyzes call content in real time and responds quickly to potential fraud. The system monitors call content as soon as the user initiates a call and uses a generative AI model to detect specific keywords and phrases to determine the likelihood of fraud. If fraud is likely, the system records the call and sends a warning to designated contacts.

[0494] This system is implemented using the following hardware and software:

[0495] Hardware: Servers, user devices (smartphones, landlines, etc.)

[0496] Software: speech recognition modules (e.g., Google Cloud Speech-to-Text), generative AI models (e.g., OpenAI GPT-4)

[0497] Call detection

[0498] When a user starts a call on the phone, the server receives a call start signal from the user terminal and detects the start of the call. For example, when an elderly person starts a call on their home landline phone, the server receives the call start signal emitted from the terminal and confirms that the call has started.

[0499] Streaming of call content

[0500] After a call starts, the device streams the call audio to the server in real time. Specifically, the device's built-in microphone converts the audio into digital data and sends that data to the server via the Internet. For example, a smartphone's microphone picks up the audio, digitizes it, and sends it to the server in real time as packets.

[0501] Voice Recognition

[0502] The server passes the received voice data to a speech recognition module (e.g., Google Cloud Speech-to-Text) and converts the voice into text. For example, if a user says "hello" during a phone conversation, the word will be converted into text as "Konnichiwa."

[0503] Text analytics

[0504] The generation AI analyzes the text data sent from the voice recognition module and detects specific keywords and phrases (e.g., "It's me," "I need money urgently," etc.). For example, if the call text contains the phrase "Please transfer the money immediately," the generation AI will detect this and calculate a score indicating the likelihood of fraud.

[0505] Determining the likelihood of fraud

[0506] The generative AI evaluates the likelihood of fraud based on the analysis results and returns the result to the server. For example, if a specific fraud-related phrase is detected, a high fraud risk score is set and the score is sent to the server.

[0507] Call recording and storage

[0508] The server records calls that are deemed likely to be fraudulent and stores the data in secure data storage. For example, recording begins the moment the user responds to the fraudster by saying, "I'll transfer the money." The recording data is then securely stored in encrypted storage.

[0509] Sending notifications

[0510] The server notifies the user and designated contacts that a potentially fraudulent call has been detected. Notifications can be sent in the form of push notifications, emails, SMS, etc. For example, a warning message stating "Your mother's call may be fraudulent" could be sent to the user's family.

[0511] As described above, the present invention aims to prevent fraud by analyzing the contents of calls in real time, and if there is a possibility of fraud, quickly recording the contents of the call and making appropriate notifications, thereby preventing fraud damage from occurring.

[0512] Prompt Sentence Examples

[0513] Examples of prompts to input to a generative AI model include:

[0514] Please explain in natural language the programming steps of the generative AI that analyzes user conversations in real time and detects specific keywords and phrases. Please also provide specific examples.

[0515] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0516] System program processing flow

[0517] Step 1: Call detection

[0518] Input: Signal to start a call

[0519] Processing: The server receives a call initiation signal from the user terminal to detect that the user is about to initiate a call.

[0520] Specific operation: For example, when an elderly person starts a call using their home landline, the device detects this and sends a signal to the server, which then receives the signal and confirms that the call has started.

[0521] Output: Call start confirmation information

[0522] Step 2: Stream the call

[0523] Input: Call audio data

[0524] Processing: The device streams the call audio to the server in real time, converts the call audio into digital data, and sends the data over the internet to the server.

[0525] How it works: For example, a smartphone's microphone picks up the audio of a call and converts it into digital data, which is then split into packets and streamed to a server.

[0526] Output: Digitized voice data

[0527] Step 3: Voice Recognition

[0528] Input: Digitized voice data

[0529] Processing: The server passes the received voice data to a speech recognition module (e.g., Google Cloud Speech-to-Text) to convert the voice into text.

[0530] Specific operation: For example, if the voice data contains "Hello", it will be converted into text data as "Konnichiwa".

[0531] Output: Text data

[0532] Step 4: Text Analysis

[0533] Input: Text data

[0534] Processing: The generative AI analyzes the text data sent from the speech recognition module to detect specific fraud-related keywords and phrases (e.g., "It's me," "I need money urgently," etc.).

[0535] Specific behavior: For example, if the text data contains the phrase "Please transfer the money immediately," the generative AI will detect this and calculate a score indicating the likelihood of fraud.

[0536] Output: Keyword analysis results and fraud likelihood score

[0537] Step 5: Determine the likelihood of fraud

[0538] Input: Keyword analysis results and fraud likelihood score

[0539] Processing: The generative AI assesses the likelihood of fraud based on the analyzed text data and score.

[0540] Specific behavior: For example, if the fraud likelihood score is set high, the generation AI will return that score to the server.

[0541] Output: Fraud likelihood assessment result

[0542] Step 6: Record and save the call

[0543] Input: Fraud likelihood assessment results

[0544] Processing: The server records calls that are determined to be likely fraudulent and stores the data in secure storage.

[0545] Specific operation: For example, the moment the user responds to the scammer by saying "I'll transfer the money," recording begins, and the recording data is stored in encrypted storage.

[0546] Output: Recorded and securely stored data

[0547] Step 7: Sending notifications

[0548] Input: Fraud likelihood assessment results and audio recording

[0549] Action: The server notifies the user and designated contacts that a potentially fraudulent call has been detected.

[0550] Specific behavior: Notifications can be sent in the form of push notifications, emails, SMS, etc. For example, a warning message stating "Your mother's call may be fraudulent" is sent to the user's family.

[0551] Output: Information message

[0552] Through these steps, the system can analyze call content in real time, quickly detect potential fraud, and take necessary action.

[0553] (Application example 1)

[0554] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0555] In modern society, telephone fraud continues to affect many people. Elderly and other vulnerable groups are particularly susceptible to fraud, easily falling prey to the sophisticated tactics of criminals. Conventional methods have made it difficult to monitor phone conversations in real time, quickly identify potential fraud, and respond accordingly. There is also a need for a system that can quickly notify a third party of potentially fraudulent calls and take appropriate action. Given this background, there is an urgent need to develop a system that can analyze potential fraud in real time during a call and take appropriate action.

[0556] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0557] In this invention, the server includes means for detecting the start of a call, means for streaming the call audio to the server in real time, means for converting the audio to text, means for analyzing the text to determine the possibility of fraud, means for recording the call if fraud is likely, means for sending a notification if fraud is likely, means for including a generative AI model for analyzing the text of the call, and means for sending a notification of suspected fraud to a notification recipient if fraud is likely. This enables fraudulent activity during a call to be detected in real time and appropriate action to be taken promptly.

[0558] The "means for detecting the start of a call" is a device or software that has the function of recognizing and notifying that a call has started.

[0559] "Means for streaming the audio of a call to a server in real time" refers to a device or software that has the function of converting audio data during a call into a digital format in real time and transmitting it to a server.

[0560] A "means for converting speech to text" is a device or software that has the function of analyzing speech data and converting it into corresponding text data.

[0561] A "means for analyzing text to determine likelihood of fraud" is a device or software capable of examining text data to identify fraud-related keywords and phrases and assess the likelihood of fraud.

[0562] "Means for recording telephone conversations when fraud is likely" refers to devices or software that have the function of automatically recording telephone conversations when fraud is deemed to be likely.

[0563] A "means for sending notifications in the event of potential fraud" is a device or software capable of sending a warning message or notification to a specified recipient in the event of potential fraud.

[0564] A "means including a generative AI model for analyzing the text of a call" is a device or software that uses generative AI to convert voice data into text and analyzes that text to detect specific keywords or phrases.

[0565] "Means for sending a notice of suspected fraud to a notification recipient when there is a high possibility of fraud" refers to a device or software that has the function of notifying a pre-determined notification recipient of suspected fraud when it is determined that there is a high possibility of fraud.

[0566] The present invention provides a system that analyzes the contents of calls in real time and responds quickly when there is a possibility of fraud. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail the preferred embodiments of the present invention.

[0567] System Configuration

[0568] The system is realized using the user's smartphone, a server, and a generative AI model.

[0569] Hardware and software used

[0570] Hardware: Smartphones, servers

[0571] Software: Android / iOS SDK, Google Cloud Speech-to-Text API, Fraud Detection AI Model

[0572] System Operation

[0573] 1. Call detection:

[0574] The user's smartphone detects the start of a call, which is notified using the phone application's integration API.

[0575] 2. Streaming Call Content:

[0576] During a call, the smartphone streams the audio in real time to a server, using the smartphone's built-in microphone to convert the audio data into a digital format and send it to the server.

[0577] 3. Speech Recognition:

[0578] The server converts the received voice data into text using the Google Cloud Speech-to-Text API.

[0579] 4. Text Analysis:

[0580] The server uses a generative AI model (e.g., based on BERT or GPT-4) to analyze the text data and detect keywords and phrases related to fraud.

[0581] 5. Determining the likelihood of fraud:

[0582] The server scores the likelihood of fraud based on the keywords and phrases detected.

[0583] 6. Recording and storage of calls:

[0584] If the server determines that there is a high possibility of fraud, it will automatically record the call and store it in secure storage (e.g., Amazon S3).

[0585] 7. Sending Notifications:

[0586] If there is a high possibility of fraud, the server will send a warning message or a notification to the user and pre-specified notification destinations using a notification API such as Twilio.

[0587] Specific examples

[0588] For example, if an elderly parent receives a call from a scammer, here's how it works:

[0589] 1. An elderly parent's smartphone detects the start of a call.

[0590] 2. The call content is sent to the server in real time.

[0591] 3. The server converts the speech into text through speech recognition.

[0592] 4. The generative AI model detects fraudulent keywords such as "It's me" and "I need money."

[0593] 5. If a call is deemed likely to be fraudulent, the call will be recorded and a warning will be sent to the user and their family.

[0594] 6. A warning message will be sent, for example, a notification saying "Your mother's call may be fraudulent."

[0595] Prompt Sentence Examples

[0596] "I want to create an app to protect against fraud. The app will analyze phone calls in real time, detect specific fraudulent keywords and phrases, and issue a warning. The prompt might look like this:

[0597] Please analyze this call to see if it contains any scam phrases, and if so, please let us know what they are."

[0598] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0599] Step 1:

[0600] The smartphone detects the start of a call

[0601] Specific operation: When a user starts a call using the phone app, a call start event occurs, and the start of the call is notified to the server via the smartphone's collaboration API. This event is sent to the server as call log information.

[0602] Input: Signal to start a call

[0603] Output: Call start notification and call log information

[0604] Step 2:

[0605] The smartphone streams the call content to the server in real time.

[0606] What it does: It uses the smartphone's microphone to capture the audio during a call and streams it to a server as digital audio data, which is sent in packets.

[0607] Input: Call audio

[0608] Output: Streaming digital audio data

[0609] Step 3:

[0610] The server passes the voice data to the speech recognition module and converts it into text.

[0611] What it does: It passes the received digital audio data to the Google Cloud Speech-to-Text API, which converts the audio into text, which is then used in the next analysis step.

[0612] Input: Digital audio data

[0613] Output: Text data

[0614] Step 4:

[0615] The server uses a generative AI model to analyze the text data and detect fraudulent keywords and phrases.

[0616] How it works: Using a generative AI model (such as BERT or GPT-4) to detect specific keywords or phrases in text data, such as "It's me" or "I need money urgently."

[0617] Input: Text data

[0618] Output: Fraudulent keyword and phrase detection results

[0619] Step 5:

[0620] The server scores the likelihood of fraud based on detected keywords and phrases.

[0621] What it does: Based on the keywords and phrases detected, it applies an algorithm to assess the likelihood of fraud and calculates a fraud risk score. It then determines the risk of the call.

[0622] Input: Fraudulent keyword or phrase detection results

[0623] Output: Fraud risk score

[0624] Step 6:

[0625] The server records potentially fraudulent calls and stores them in secure storage.

[0626] What it does: If the fraud risk score is determined to be high, the system will begin recording the call and store the recording in secure cloud storage such as AWS S3. After recording is complete, the data is encrypted and stored.

[0627] Input: Fraud Risk Score

[0628] Output: Save call recording data

[0629] Step 7:

[0630] Sends a warning notification if the server is at high fraud risk

[0631] Specific behavior: If the fraud risk score is high, a warning message is sent to the user and pre-designated notification recipients. Notifications such as "Possible fraud" are sent via push notification, email, or SMS using notification APIs such as Twilio.

[0632] Input: Fraud Risk Score

[0633] Output: Sending a warning message

[0634] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0635] The present invention combines a system that analyzes call content in real time and responds quickly to potential fraudulent activity with an emotion engine that recognizes user emotions. The system monitors call content from the moment the user initiates a call and analyzes specific keywords, phrases, and even voice characteristics to more accurately determine the likelihood of fraud. If a fraudulent activity is suspected, the system records the call and sends a warning to designated notification recipients.

[0636] System Configuration

[0637] 1. Call detection

[0638] The server waits for and receives a call start signal from the terminal to detect when the user has started a call on the phone, and when the server receives this signal, it starts call monitoring in real time.

[0639] 2. Streaming of call content

[0640] The terminal streams the voice to the server in real time when the user starts a call. The terminal has the function of converting the voice during the call into digital data and sending this data to the server.

[0641] 3. Voice Recognition

[0642] The server passes the received voice data to a speech recognition module, which converts the voice into text using a pre-trained model.

[0643] 4. Text Analysis

[0644] The generative AI analyzes the text data sent from the voice recognition module to detect specific fraud patterns and keywords, while also analyzing voice characteristics using an emotion engine to recognize the user's emotions.

[0645] 5. Determining the likelihood of fraud

[0646] The generative AI and emotion engine combine the results of their analysis to determine the likelihood of fraud. The generative AI calculates the probability of fraud for each keyword or phrase, and then takes into account the results of the emotion engine's analysis of user sentiment to precisely assess the likelihood of fraud.

[0647] 6. Recording and saving of call content

[0648] The server records calls that are deemed likely to be fraudulent and stores the data in secure storage, which can then be used for later review or as evidence.

[0649] 7. Sending Notifications

[0650] When a potentially fraudulent call is detected, the server sends a warning to the user and designated contacts (family and relatives). Notification methods can be selected from push notifications, emails, SMS, etc.

[0651] A natural language description of the program's processing

[0652] Call detection

[0653] When a user starts a call, the server detects the call start event and prepares for real-time analysis. For example, when an elderly person starts a call on their home landline phone, the server automatically detects the call.

[0654] Streaming of call content

[0655] The device transmits the call audio to the server in real time. The device is equipped with a microphone that converts the audio into digital data and sends it to the server as packets.

[0656] Voice Recognition

[0657] The server passes the received voice data to a speech recognition module, which converts the voice data into text data that is used in further analysis steps.

[0658] Text and Sentiment Analysis

[0659] The generative AI analyzes the text data sent from the voice recognition module to detect specific keywords and phrases. At the same time, the emotion engine analyzes the voice characteristics to recognize the user's emotions. For example, the emotion engine analyzes the tone, speed, and volume of the user's voice to determine stress or urgency.

[0660] Determining the likelihood of fraud

[0661] Generative AI combines the results of analytics and sentiment analysis to assess the likelihood of fraud. This process involves scoring the fraud risk for each keyword or phrase, and then taking into account the results of the sentiment engine to make a final decision.

[0662] Call recording and storage

[0663] The server records any audio data that the AI ​​determines to be fraudulent and stores it in secure storage. For example, recording begins when a phrase such as "transfer your money now" is detected.

[0664] Sending notifications

[0665] The server will notify the user and the designated contacts that a potentially fraudulent call has been detected. The notification will be sent as a warning message saying, "Your mother's call may be fraudulent."

[0666] In this way, the present invention provides a system that analyzes call content in real time and combines it with an emotion engine to more accurately determine the possibility of fraud and respond quickly. Specifically, the system automatically starts working the moment the user starts a call, and if a possible fraud is detected, a notification is sent immediately, preventing fraud damage before it occurs.

[0667] The processing flow will be explained below.

[0668] Step 1:

[0669] The server detects when a user starts a call on the phone. The server receives a call start signal from the user's device and prepares for real-time call monitoring. Specifically, the server receives a signal from the device as soon as the call starts and starts the analysis module.

[0670] Step 2:

[0671] When a user initiates a call, the device streams the call audio to the server in real time. The device digitizes the audio picked up through the microphone and sends the data to the server. For example, audio data from a landline phone or smartphone is continuously sent to the server.

[0672] Step 3:

[0673] The server passes the received voice data to a speech recognition module, which converts it into text data. The speech recognition module uses a pre-trained model to analyze the voice and convert it into corresponding text, which is used in the next analysis step.

[0674] Step 4:

[0675] The generation AI receives the text data sent from the speech recognition module and begins analyzing it to detect specific fraud patterns and keywords. At the same time, the emotion engine analyzes this voice data to recognize the user's emotions. For example, the generation AI detects fraud-specific phrases such as "I need money urgently," and the emotion engine analyzes the user's tone and speed of voice to indicate feelings of tension or impatience.

[0676] Step 5:

[0677] The generative AI and emotion engine combine the analysis results to determine the likelihood of fraud. First, the generative AI calculates a fraud score for the detected keywords and phrases, and then the emotion engine makes a final assessment based on the user's emotion results identified by the generative AI. If it determines that there is a high likelihood of fraud, it returns the result to the server.

[0678] Step 6:

[0679] The server then issues instructions to record calls deemed likely to be fraudulent and store the data in secure storage, where it can later be used for verification or evidence. This step is triggered if specific phrases, such as "I need money now," are detected.

[0680] Step 7:

[0681] If the server detects a potentially fraudulent call, it will send a warning to the user and designated recipients (for example, family and relatives). Notification methods can be selected from push notifications, emails, SMS, etc. For example, a warning message stating, "There is a possibility that your mother's call may be fraudulent" will be sent.

[0682] Step 8:

[0683] The user and the designated contacts receive the warning from the server and check the details. The contacts, such as family members, check the warning message and, if necessary, contact the elderly person again to check the details of the call. At this time, they can also notify the police if necessary and take detailed measures.

[0684] In this way, the present invention analyzes the content of phone calls in real time and combines it with an emotion engine to create a system that can accurately determine the possibility of fraud and respond quickly, thereby preventing fraud damage before it occurs.

[0685] Example 2

[0686] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0687] As fraudulent phone calls increase, there is a problem that vulnerable people, such as the elderly, are particularly susceptible to these attacks. Conventional methods have not established a means to detect fraud during a call and respond quickly. Furthermore, there is a need for a highly accurate analysis method that takes into account the user's emotions.

[0688] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for detecting the start of a call, means for streaming the voice of the call to the central processing unit in real time, means for converting the voice to text, means for analyzing the text to determine the possibility of fraud and the user's emotions, means for recording the contents of the call if there is a high possibility of fraud, and means for sending a notification if there is a possibility of fraud. This makes it possible to accurately determine the possibility of fraud during a call and respond quickly.

[0689] The "means for detecting the start of a call" is a function that automatically recognizes the timing when the user starts a call and notifies the system of the start of the call.

[0690] "Means for streaming the voice of a call to a central processing unit in real time" refers to a function that converts the voice of a call into digital data and instantly transmits that data to a central processing unit.

[0691] The "means for converting voice to text" is a function that analyzes received voice data and converts it into text data.

[0692] The "means for analyzing text to determine the likelihood of fraud and the user's emotions" is a function that uses a generative language model and an emotion analysis device to evaluate the likelihood of fraud and the user's emotions from text data.

[0693] "Means for recording phone calls when there is a high risk of fraud" refers to a function that records the contents of phone calls as digital data when text analysis and sentiment analysis indicate a high risk of fraud.

[0694] "Means for sending notifications in the event of possible fraud" is a function that sends a warning message to the user and designated notification recipients when a risk of fraud is detected.

[0695] A "generative language model" is an algorithm that has been trained in advance on a large amount of language data, and is used to analyze input text data and detect specific patterns and keywords.

[0696] An "emotion analysis device" is a software or hardware device that recognizes a user's emotions from voice data or text data.

[0697] The present invention combines a system that analyzes call content in real time and responds quickly to potential fraud, with an emotion engine that recognizes user emotions. Specific embodiments of this system are described below.

[0698] The system mainly consists of the following elements:

[0699] server

[0700] Terminal

[0701] Voice Recognition Module

[0702] Generation AI

[0703] Emotion Engine

[0704] Secure Storage

[0705] Notification System

[0706] The server detects when a user starts a call by receiving a signal from the terminal. Upon receiving the call start signal, the server starts real-time analysis for call monitoring. When the terminal starts a call, it converts the voice data into digital data and streams it to the server in real time.

[0707] The speech recognition module converts the voice data received by the server into text data. This uses a generative AI model that has been trained on a large amount of data in advance. The generative AI model converts voice to text with high accuracy, removes background noise, and identifies the speaker.

[0708] The generative AI analyzes the text data sent from the speech recognition module to detect specific keywords and phrases. At the same time, the emotion engine analyzes voice characteristics (tone, speed, volume, etc.) to recognize the user's emotions. For example, if the user's voice sounds tense or urgent, the emotion engine will determine the level of stress or urgency.

[0709] Based on the results of this analysis, the generative AI performs a process to assess the likelihood of fraud. It scores the risk of fraud and integrates the results of the emotion engine to make a final judgment. For example, if a phrase such as "Please tell me the bank details" is detected and the tone of voice used is unstable, it will be assessed as having a high likelihood of fraud.

[0710] If the AI ​​generator determines that a call is likely to be fraudulent, the server records the call and stores the data in secure storage, which has access restrictions and can be accessed as needed.

[0711] Furthermore, when a potentially fraudulent call is detected, the server sends a warning to the user and designated contacts. Notification methods include push notification, email, and SMS. For example, a warning message stating, "Your mother's call may be fraudulent" is sent to the contact.

[0712] For example, if an elderly person utters the phrase "Please transfer the money now" during a call, the system will immediately detect this phrase and determine that it is likely a fraud. In this case, the server will record the call, store the data in secure storage, and send an alert to the designated notification recipient.

[0713] Examples of prompts include:

[0714] "Design a fraud detection and notification system when an elderly person utters a phrase that could be fraudulent during a call. The system should include a server that analyzes the call audio in real time, evaluates the possibility of fraud, and records the call and sends a notification if necessary."

[0715] In this way, by analyzing the content of phone calls in real time and combining it with an emotion engine, the present invention can accurately determine the possibility of fraud and respond quickly. The system operates automatically from the moment the user starts a call, and if a possible fraud is detected, a notification is sent immediately, preventing fraud damage before it occurs.

[0716] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0717] Processing Steps

[0718] Step 1:

[0719] The server detects that a user has started a call. It receives a call start signal from the terminal as input and processes this signal as a call start event. This causes the server to start preparing for call monitoring and prepare for real-time voice data analysis. Specifically, when the server receives the call start signal, it starts the analysis module.

[0720] Step 2:

[0721] The terminal converts the call voice into digital data and streams it to the server in real time. As input, the user's voice is collected through a microphone and the voice data is converted into digital data. As output, this digital data is divided into packets and sent to the server. Specifically, the voice conversion module of the terminal generates digital voice data, and the network module sends it to the server.

[0722] Step 3:

[0723] The server passes the received voice data to a voice recognition module, which converts the voice into text data. As input, the digital voice data transmitted in real time is passed to the voice recognition module. As output, this voice data is converted into text data, which is used for the next analysis step. Specifically, the voice recognition module analyzes the voice data using a generative AI model and generates text data.

[0724] Step 4:

[0725] The generation AI analyzes the text data sent from the voice recognition module to detect specific keywords and phrases. At the same time, the emotion engine analyzes the voice characteristics and recognizes the user's emotions. Text data and voice characteristics data are provided as input. The output is an assessment of the likelihood of fraud and the user's emotional state. Specifically, the generation AI checks for the presence of specified keywords and phrases, and the emotion engine analyzes the tone, speed, and volume of the voice.

[0726] Step 5:

[0727] Generative AI integrates the results of analysis and sentiment analysis to assess the likelihood of fraud. The inputs are the detection results of specific keywords and phrases and the analysis results of the sentiment engine. The output is a fraud risk score and a final judgment. Specifically, a risk score is calculated for each keyword or phrase, and the results of sentiment analysis are then added to make the final judgment.

[0728] Step 6:

[0729] If the generation AI determines that there is a high possibility of fraud, the server records the call and stores the data in secure storage. The input is call data with a high fraud risk assessment. The output is the recorded voice data, which is encrypted and stored. Specifically, the call is recorded in real time, and the data is sent to secure storage via an encryption module.

[0730] Step 7:

[0731] The server notifies the user and designated contacts that a potentially fraudulent call has been detected. Event information about the call that was determined to be at high risk of fraud is provided as input. A warning message is sent as output via push notification, email, SMS, etc. Specifically, a notification API is used to send a message to the user or family member stating, "There has been a potentially fraudulent call."

[0732] This is the flow of the program processing for this system. At each step, specific data processing and calculations are performed, allowing for rapid response in cases where there is a high possibility of fraud.

[0733] (Application example 2)

[0734] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0735] In recent years, telephone fraud has been on the rise, with elderly people being particularly vulnerable to its fallout. Because such fraudulent activities use sophisticated persuasive and psychological tactics, it is difficult to ensure sufficient detection accuracy by simply analyzing keywords and phrases in phone call content. Furthermore, even if a possible fraud is detected, it is difficult to prevent damage without a rapid response. Therefore, a system is needed that can accurately determine the possibility of fraud and respond quickly by taking into account the user's emotional state in addition to real-time analysis of phone call content.

[0736] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for detecting the start of a call, means for streaming the call audio to the server in real time, means for converting the audio into text, and means for analyzing the text to determine the possibility of fraud. This enables real-time analysis of the call content. The system also includes means for recording the call content when there is a high possibility of fraud, means for sending a notification when there is a possibility of fraud, and an emotion engine for analyzing the user's emotions in real time, and is equipped with means for precisely evaluating the possibility of fraud by integrating the generative AI model and the emotion engine. This improves the accuracy of determining the possibility of fraud and enables prompt prevention of damage.

[0737] The "means for detecting the start of a call" is a function for automatically detecting the moment when a user starts a call on the telephone.

[0738] "Means for streaming the voice of a call to a server in real time" refers to a function that digitizes the voice data during a call in real time and transmits it to a server via the Internet.

[0739] "Means for converting voice to text" refers to voice recognition technology that analyzes received voice data and converts it into text data.

[0740] "Means for analyzing text to determine the possibility of fraud" refers to a function that analyzes text data converted from speech and detects specific fraudulent keywords and phrases.

[0741] "Means for recording phone calls when there is a high possibility of fraud" refers to a function that records and saves as a digital audio file any phone call that is determined to be highly likely to be fraudulent as a result of analysis.

[0742] "Means for sending notifications in the event of possible fraud" is a function that sends a warning to the user and designated notification recipients when a call with a high risk of fraud is detected.

[0743] The "emotion engine that analyzes user emotions in real time" is a technology that analyzes voice characteristics (e.g., tone, speed, volume) during a call and recognizes the user's emotional state (e.g., stress, urgency) in real time.

[0744] A "generative AI model" is an artificial intelligence model that has been trained in advance with large amounts of data, and is a technology used to detect specific keywords and phrases with high accuracy.

[0745] "Means of precisely assessing the likelihood of fraud by integrating an emotion engine" is a function that integrates the text analysis results of the generative AI model with the emotion analysis results of the emotion engine to comprehensively determine the likelihood of fraud and improve its accuracy.

[0746] The "means for receiving notifications and allowing users to check the content of fraudulent calls" is an interface function that allows users to check the warning notifications they receive and view the details of calls suspected to be fraudulent.

[0747] System program generation

[0748] Hardware and Software Use

[0749] The system for implementing this invention mainly consists of a server and a user's smartphone. The server is equipped with a high-speed processor and a large-capacity SSD, and runs a speech recognition model (e.g., Google Speech-to-Text), a sentiment analysis engine (e.g., IBM Watson Tone Analyzer), and a generative AI model (e.g., OpenAI's GPT-4).

[0750] A smartphone uses the following hardware and software:

[0751] Microphone: Captures call audio.

[0752] Communications module: Streams call audio to the server in real time.

[0753] Internet connection: Allows for high speed transmission of data.

[0754] A natural language description of the program's processing

[0755] Detecting the start of a call

[0756] When a user starts a call on the phone, the smartphone application detects the call start event and notifies the server, which receives the notification and begins preparing for real-time analysis.

[0757] Streaming call audio

[0758] The user's smartphone picks up the call audio with a microphone, converts it into digital data, and sends this data over the Internet to a server, which receives the audio streaming data in real time.

[0759] Speech-to-text

[0760] The server converts the received voice data into text using a speech recognition model (e.g., Google Speech-to-Text), which is then used for subsequent analysis.

[0761] Text and Sentiment Analysis

[0762] A server-based generative AI model (e.g., GPT-4) analyzes the text data obtained from the speech recognition module to detect specific fraudulent keywords and phrases. At the same time, a sentiment analysis engine (e.g., IBM Watson Tone Analyzer) analyzes voice characteristics to recognize the user's emotions in real time. This process is carried out using advanced algorithms.

[0763] Determining the likelihood of fraud

[0764] The server combines the results of the generative AI model and the sentiment analysis engine to precisely assess the likelihood of fraud. It scores the fraud risk by combining specific keywords and the user's emotional state (e.g., high stress). If it determines there is a high likelihood of fraud, the server records the call and stores it in secure storage.

[0765] Warning notice

[0766] The server sends a warning to the user and pre-designated notification recipients (e.g., family members) for calls that are determined to be highly likely to be fraudulent. This notification can be done by push notification, email, SMS, or other means, and will send a message such as, "Signs of fraud have been detected in the content of your call."

[0767] Examples of concrete examples and prompts

[0768] When an elderly person receives a scam call pretending to be their grandchild, the emotion analysis engine detects that the tone and speed of the elderly person's voice warrants attention. The generative AI model picks up phrases such as "transfer money." This causes the server to assess the fraud risk as high, and recording begins. A notification is also sent to the family saying, "There is a possibility that your mother's call is a scam."

[0769] Prompt Sentence Examples

[0770] "Detect potential fraudulent elements in the following phone calls."

[0771] "Assess the user's stress level based on tone of voice and emotional analysis."

[0772] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0773] Step 1:

[0774] Detect the start of a call

[0775] Input: The user initiates a call.

[0776] The server receives a call start event from the smartphone application and detects the start of a call, which allows the server to prepare for real-time analysis.

[0777] Output: Call start information.

[0778] Specific behavior: When a user starts a call on their smartphone, the application catches the event and sends a call start signal to the server.

[0779] Step 2:

[0780] Streaming call audio

[0781] Input: Smartphone call audio.

[0782] The smartphone picks up the call audio through a microphone, converts it into digital data, and streams this data to a server over the internet.

[0783] Output: Real-time audio data.

[0784] How it works: The smartphone's microphone picks up sound, converts the sound signal into digital packets in real time, and sends them to a server. Because this data is sent in real time, it can be analyzed without delay.

[0785] Step 3:

[0786] Speech-to-text

[0787] Input: Real-time audio data.

[0788] The server sends the received voice data to a speech recognition model (e.g., Google Speech-to-Text) and converts the voice into text.

[0789] Output: Text data.

[0790] How it works: A server-based speech recognition model analyzes the audio data and converts it into corresponding text data, which is then used in subsequent analysis steps.

[0791] Step 4:

[0792] Text and Sentiment Analysis

[0793] Input: Text data of speech recognition results.

[0794] A generative AI model (e.g., GPT-4) analyzes text data to detect specific fraudulent keywords and phrases, while an emotion engine (e.g., IBM Watson Tone Analyzer) analyzes voice characteristics to assess user sentiment in real time.

[0795] Output: Deception probability and emotional state assessment results.

[0796] How it works: A generative AI model examines text to identify keywords and phrases related to fraud, while an emotion engine analyzes the tone, rate, and volume of speech to identify emotional states.

[0797] Step 5:

[0798] Determining the likelihood of fraud

[0799] Input: Deception likelihood and emotional state assessment results.

[0800] The server combines the results of the generative AI model and the emotion engine to assess the likelihood of fraud, and if the fraud risk is determined to be high, the call is recorded.

[0801] Output: Fraud risk score and audio recording.

[0802] How it works: The server combines the results of text analysis and sentiment analysis to score the risk of fraud. If the score exceeds a threshold, the call is recorded and stored in a secure location.

[0803] Step 6:

[0804] Warning notice

[0805] Input: Fraud risk score and audio recording.

[0806] The server will send a warning to the user and designated contacts if fraud is likely.

[0807] Output: A warning message.

[0808] What happens: The server uses the notification system to send a warning message to the user and their family members via push notification, email, SMS, etc., such as "Possible fraud detected."

[0809] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0810] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0811] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0812] [Third embodiment]

[0813] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0814] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0815] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0816] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0817] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0818] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0819] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0820] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0821] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0822] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0823] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0824] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0825] This invention is a system that analyzes call content in real time and responds quickly to potential fraud. The system monitors call content when a user initiates a call and uses generative AI to detect specific keywords and phrases to determine the possibility of fraud. If fraud is likely, the system records the call and sends a warning to designated notification recipients.

[0826] System Configuration

[0827] 1. Call detection

[0828] The server detects when a user initiates a call on the telephone by receiving a call initiation signal from the user terminal.

[0829] 2. Streaming of call content

[0830] The device streams the call audio to the server in real time. The device has a microphone that converts the audio into digital data and transmits the data to the server.

[0831] 3. Voice Recognition

[0832] The server passes the received voice data to a speech recognition module that converts the voice data to text, which uses a pre-trained model to convert speech to text with high accuracy.

[0833] 4. Text Analysis

[0834] Generative AI analyzes the converted call content to detect specific fraud patterns and keywords, using algorithms to assess the likelihood of fraud.

[0835] 5. Determining the likelihood of fraud

[0836] The generation AI determines whether there is a possibility of fraud based on the analysis results, and if there is a high possibility of fraud, it returns the result to the server.

[0837] 6. Recording and saving of call content

[0838] If the generation AI determines that there is a high possibility of fraud, the server will record the call and store it in secure data storage.

[0839] 7. Sending Notifications

[0840] If the server detects a potentially fraudulent call, it will send a warning to the user and their designated contacts via push notification, email, SMS, or other methods.

[0841] A natural language description of the program's processing

[0842] Call detection

[0843] When a user starts a call, the server detects the call start event and prepares for real-time analysis. For example, when an elderly person starts a call on their home landline phone, the server automatically detects the call.

[0844] Streaming of call content

[0845] The terminal transmits the voice during the call to the server in real time. The actual voice is converted into digital data and sent as packets to the server.

[0846] Voice Recognition

[0847] The server analyzes the received voice data and converts it into text using a speech recognition module, which is then used in the next analysis step.

[0848] Text analytics

[0849] The generation AI analyzes the text data received from the voice recognition module and detects specific keywords and phrases to determine whether the data is fraudulent. For example, if the data contains phrases such as "It's me" or "I need money urgently," the generation AI will detect this.

[0850] Determining the likelihood of fraud

[0851] The generative AI evaluates the likelihood of fraud based on the analysis results and returns the score to the server, flagging cases where fraud is highly likely.

[0852] Call recording and storage

[0853] The server records calls that are deemed likely to be fraudulent and stores the data in secure storage. For example, recording begins when a phrase such as "I want to transfer money" is detected.

[0854] Sending notifications

[0855] The server notifies the user and the designated contacts that a potentially fraudulent call has been detected. The notification may be sent as a warning message such as, "Your mother's call may be fraudulent."

[0856] In this way, the present invention aims to prevent fraud by analyzing the contents of calls in real time, and if there is a possibility of fraud, quickly recording the contents of the call and making appropriate notifications, thereby preventing fraud damage from occurring.

[0857] The processing flow will be explained below.

[0858] Step 1:

[0859] The server waits for and receives a call start signal from the terminal to detect when the user has started a call on the phone, and when the server receives this signal, it starts monitoring the call in real time.

[0860] Step 2:

[0861] The device streams the audio to the server in real time as soon as the user starts a call. The device has a microphone that converts the audio into digital data and sends it to the server.

[0862] Step 3:

[0863] The server passes the received voice data to a speech recognition module, which analyzes the voice data and converts it into corresponding text data in real time, ready for the subsequent analysis step.

[0864] Step 4:

[0865] The Generative AI receives the text data sent from the speech recognition module and begins analyzing it to detect specific fraud patterns and keywords. Using pre-trained models, the Generative AI scores the likelihood of fraud.

[0866] Step 5:

[0867] The generation AI determines whether there is a possibility of fraud based on the analysis results. The generation AI calculates the probability of fraud for each keyword or phrase and returns the result to the server as a score. If there is a high possibility of fraud, it issues instructions to the server.

[0868] Step 6:

[0869] If the server determines that a call is likely to be fraudulent, it automatically records the call audio and stores the recorded audio in secure data storage for later review.

[0870] Step 7:

[0871] When a potentially fraudulent call is detected, the server sends a warning to the user and designated contacts (family and relatives) via push notification, email, SMS, etc.

[0872] Step 8:

[0873] The user and the designated contacts will receive a warning from the server. If the user or their family members determine that the information is likely fraudulent, they can take appropriate action, such as further investigating the matter or reporting it to the police.

[0874] For example, when an elderly person starts a call on their home landline, the server automatically detects the call and analyzes the content in real time. If a phrase such as "I need money urgently" is detected during the call, the server records the call and simultaneously sends a warning to the family. By receiving this warning, the family can prevent fraud before it occurs.

[0875] Example 1

[0876] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0877] In recent years, telephone fraud has been on the rise, with many victims reported, particularly among the elderly. To prevent such fraud, a system is needed that can monitor call content in real time and quickly detect possible fraud. However, current systems have difficulty analyzing call content in real time and accurately determining the possibility of fraud. Therefore, there is a need for a method that can accurately detect possible fraud during a call and respond quickly.

[0878] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0879] In this invention, the server includes means for detecting the start of a call, means for streaming the call audio to the server in real time, means for converting the audio into text, means for analyzing the text using a model for detecting specific keywords and phrases to determine the possibility of fraud, means for recording the content of the call and saving it in secure data storage if there is a high possibility of fraud, means for sending a notification if there is a possibility of fraud, and means for receiving the notification and allowing the user to confirm the content of the fraudulent call. This makes it possible to detect the possibility of fraud during a call with high accuracy in real time and take necessary measures quickly.

[0880] A "means for detecting the start of a call" is a device or system for detecting the moment when a user starts a telephone call.

[0881] "Means for streaming telephone call audio to a server in real time" refers to a device or system for converting telephone call audio into digital data in real time and transmitting it to a server via the Internet.

[0882] A "means for converting speech to text" is a device or system that uses speech recognition technology to analyze received speech data and convert it into text data.

[0883] A "model for detecting specific keywords and phrases" is a trained algorithm used to analyze text data during calls and detect specific fraud-related keywords and phrases.

[0884] The "means for determining the likelihood of fraud" is a device or system that evaluates the likelihood of fraud based on the analyzed text data and provides the results.

[0885] A "call recording device" is a device or system that records calls that are determined to be likely fraudulent and stores the data.

[0886] "Secure Data Storage" means an encrypted data storage device or system for securely storing recordings.

[0887] A "means for sending a notification" is a device or system for sending an alert to designated notification contacts when a potentially fraudulent call is detected.

[0888] The "means for receiving the notification and allowing the user to check the content of the fraudulent call" refers to a device or system that receives the sent notification and allows the user to check the content of the notification.

[0889] This invention is a system that analyzes call content in real time and responds quickly to potential fraud. The system monitors call content as soon as the user initiates a call and uses a generative AI model to detect specific keywords and phrases to determine the likelihood of fraud. If fraud is likely, the system records the call and sends a warning to designated contacts.

[0890] This system is implemented using the following hardware and software:

[0891] Hardware: Servers, user devices (smartphones, landlines, etc.)

[0892] Software: speech recognition modules (e.g., Google Cloud Speech-to-Text), generative AI models (e.g., OpenAI GPT-4)

[0893] Call detection

[0894] When a user starts a call on the phone, the server receives a call start signal from the user terminal and detects the start of the call. For example, when an elderly person starts a call on their home landline phone, the server receives the call start signal emitted from the terminal and confirms that the call has started.

[0895] Streaming of call content

[0896] After a call starts, the device streams the call audio to the server in real time. Specifically, the device's built-in microphone converts the audio into digital data and sends that data to the server via the Internet. For example, a smartphone's microphone picks up the audio, digitizes it, and sends it to the server in real time as packets.

[0897] Voice Recognition

[0898] The server passes the received voice data to a speech recognition module (e.g., Google Cloud Speech-to-Text) and converts the voice into text. For example, if a user says "hello" during a phone conversation, the word will be converted into text as "Konnichiwa."

[0899] Text analytics

[0900] The generation AI analyzes the text data sent from the voice recognition module and detects specific keywords and phrases (e.g., "It's me," "I need money urgently," etc.). For example, if the call text contains the phrase "Please transfer the money immediately," the generation AI will detect this and calculate a score indicating the likelihood of fraud.

[0901] Determining the likelihood of fraud

[0902] The generative AI evaluates the likelihood of fraud based on the analysis results and returns the result to the server. For example, if a specific fraud-related phrase is detected, a high fraud risk score is set and the score is sent to the server.

[0903] Call recording and storage

[0904] The server records calls that are deemed likely to be fraudulent and stores the data in secure data storage. For example, recording begins the moment the user responds to the fraudster by saying, "I'll transfer the money." The recording data is then securely stored in encrypted storage.

[0905] Sending notifications

[0906] The server notifies the user and designated contacts that a potentially fraudulent call has been detected. Notifications can be sent in the form of push notifications, emails, SMS, etc. For example, a warning message stating "Your mother's call may be fraudulent" could be sent to the user's family.

[0907] As described above, the present invention aims to prevent fraud by analyzing the contents of calls in real time, and if there is a possibility of fraud, quickly recording the contents of the call and making appropriate notifications, thereby preventing fraud damage from occurring.

[0908] Prompt Sentence Examples

[0909] Examples of prompts to input to a generative AI model include:

[0910] Please explain in natural language the programming steps of the generative AI that analyzes user conversations in real time and detects specific keywords and phrases. Please also provide specific examples.

[0911] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0912] System program processing flow

[0913] Step 1: Call detection

[0914] Input: Signal to start a call

[0915] Processing: The server receives a call initiation signal from the user terminal to detect that the user is about to initiate a call.

[0916] Specific operation: For example, when an elderly person starts a call using their home landline, the device detects this and sends a signal to the server, which then receives the signal and confirms that the call has started.

[0917] Output: Call start confirmation information

[0918] Step 2: Stream the call

[0919] Input: Call audio data

[0920] Processing: The device streams the call audio to the server in real time, converts the call audio into digital data, and sends the data over the internet to the server.

[0921] How it works: For example, a smartphone's microphone picks up the audio of a call and converts it into digital data, which is then split into packets and streamed to a server.

[0922] Output: Digitized voice data

[0923] Step 3: Voice Recognition

[0924] Input: Digitized voice data

[0925] Processing: The server passes the received voice data to a speech recognition module (e.g., Google Cloud Speech-to-Text) to convert the voice into text.

[0926] Specific operation: For example, if the voice data contains "Hello", it will be converted into text data as "Konnichiwa".

[0927] Output: Text data

[0928] Step 4: Text Analysis

[0929] Input: Text data

[0930] Processing: The generative AI analyzes the text data sent from the speech recognition module to detect specific fraud-related keywords and phrases (e.g., "It's me," "I need money urgently," etc.).

[0931] Specific behavior: For example, if the text data contains the phrase "Please transfer the money immediately," the generative AI will detect this and calculate a score indicating the likelihood of fraud.

[0932] Output: Keyword analysis results and fraud likelihood score

[0933] Step 5: Determine the likelihood of fraud

[0934] Input: Keyword analysis results and fraud likelihood score

[0935] Processing: The generative AI assesses the likelihood of fraud based on the analyzed text data and score.

[0936] Specific behavior: For example, if the fraud likelihood score is set high, the generation AI will return that score to the server.

[0937] Output: Fraud likelihood assessment result

[0938] Step 6: Record and save the call

[0939] Input: Fraud likelihood assessment results

[0940] Processing: The server records calls that are determined to be likely fraudulent and stores the data in secure storage.

[0941] Specific operation: For example, the moment the user responds to the scammer by saying "I'll transfer the money," recording begins, and the recording data is stored in encrypted storage.

[0942] Output: Recorded and securely stored data

[0943] Step 7: Sending notifications

[0944] Input: Fraud likelihood assessment results and audio recording

[0945] Action: The server notifies the user and designated contacts that a potentially fraudulent call has been detected.

[0946] Specific behavior: Notifications can be sent in the form of push notifications, emails, SMS, etc. For example, a warning message stating "Your mother's call may be fraudulent" is sent to the user's family.

[0947] Output: Information message

[0948] Through these steps, the system can analyze call content in real time, quickly detect potential fraud, and take necessary action.

[0949] (Application example 1)

[0950] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0951] In modern society, telephone fraud continues to affect many people. Elderly and other vulnerable groups are particularly susceptible to fraud, easily falling prey to the sophisticated tactics of criminals. Conventional methods have made it difficult to monitor phone conversations in real time, quickly identify potential fraud, and respond accordingly. There is also a need for a system that can quickly notify a third party of potentially fraudulent calls and take appropriate action. Given this background, there is an urgent need to develop a system that can analyze potential fraud in real time during a call and take appropriate action.

[0952] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0953] In this invention, the server includes means for detecting the start of a call, means for streaming the call audio to the server in real time, means for converting the audio to text, means for analyzing the text to determine the possibility of fraud, means for recording the call if fraud is likely, means for sending a notification if fraud is likely, means for including a generative AI model for analyzing the text of the call, and means for sending a notification of suspected fraud to a notification recipient if fraud is likely. This enables fraudulent activity during a call to be detected in real time and appropriate action to be taken promptly.

[0954] The "means for detecting the start of a call" is a device or software that has the function of recognizing and notifying that a call has started.

[0955] "Means for streaming the audio of a call to a server in real time" refers to a device or software that has the function of converting audio data during a call into a digital format in real time and transmitting it to a server.

[0956] A "means for converting speech to text" is a device or software that has the function of analyzing speech data and converting it into corresponding text data.

[0957] A "means for analyzing text to determine likelihood of fraud" is a device or software capable of examining text data to identify fraud-related keywords and phrases and assess the likelihood of fraud.

[0958] "Means for recording telephone conversations when fraud is likely" refers to devices or software that have the function of automatically recording telephone conversations when fraud is deemed to be likely.

[0959] A "means for sending notifications in the event of potential fraud" is a device or software capable of sending a warning message or notification to a specified recipient in the event of potential fraud.

[0960] A "means including a generative AI model for analyzing the text of a call" is a device or software that uses generative AI to convert voice data into text and analyzes that text to detect specific keywords or phrases.

[0961] "Means for sending a notice of suspected fraud to a notification recipient when there is a high possibility of fraud" refers to a device or software that has the function of notifying a pre-determined notification recipient of suspected fraud when it is determined that there is a high possibility of fraud.

[0962] The present invention provides a system that analyzes the contents of calls in real time and responds quickly when there is a possibility of fraud. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail the preferred embodiments of the present invention.

[0963] System Configuration

[0964] The system is realized using the user's smartphone, a server, and a generative AI model.

[0965] Hardware and software used

[0966] Hardware: Smartphones, servers

[0967] Software: Android / iOS SDK, Google Cloud Speech-to-Text API, Fraud Detection AI Model

[0968] System Operation

[0969] 1. Call detection:

[0970] The user's smartphone detects the start of a call, which is notified using the phone application's integration API.

[0971] 2. Streaming Call Content:

[0972] During a call, the smartphone streams the audio in real time to a server, using the smartphone's built-in microphone to convert the audio data into a digital format and send it to the server.

[0973] 3. Speech Recognition:

[0974] The server converts the received voice data into text using the Google Cloud Speech-to-Text API.

[0975] 4. Text Analysis:

[0976] The server uses a generative AI model (e.g., based on BERT or GPT-4) to analyze the text data and detect keywords and phrases related to fraud.

[0977] 5. Determining the likelihood of fraud:

[0978] The server scores the likelihood of fraud based on the keywords and phrases detected.

[0979] 6. Recording and storage of calls:

[0980] If the server determines that there is a high possibility of fraud, it will automatically record the call and store it in secure storage (e.g., Amazon S3).

[0981] 7. Sending Notifications:

[0982] If there is a high possibility of fraud, the server will send a warning message or a notification to the user and pre-specified notification destinations using a notification API such as Twilio.

[0983] Specific examples

[0984] For example, if an elderly parent receives a call from a scammer, here's how it works:

[0985] 1. An elderly parent's smartphone detects the start of a call.

[0986] 2. The call content is sent to the server in real time.

[0987] 3. The server converts the speech into text through speech recognition.

[0988] 4. The generative AI model detects fraudulent keywords such as "It's me" and "I need money."

[0989] 5. If a call is deemed likely to be fraudulent, the call will be recorded and a warning will be sent to the user and their family.

[0990] 6. A warning message will be sent, for example, a notification saying "Your mother's call may be fraudulent."

[0991] Prompt Sentence Examples

[0992] "I want to create an app to protect against fraud. The app will analyze phone calls in real time, detect specific fraudulent keywords and phrases, and issue a warning. The prompt might look like this:

[0993] Please analyze this call to see if it contains any scam phrases, and if so, please let us know what they are."

[0994] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0995] Step 1:

[0996] The smartphone detects the start of a call

[0997] Specific operation: When a user starts a call using the phone app, a call start event occurs, and the start of the call is notified to the server via the smartphone's collaboration API. This event is sent to the server as call log information.

[0998] Input: Signal to start a call

[0999] Output: Call start notification and call log information

[1000] Step 2:

[1001] The smartphone streams the call content to the server in real time.

[1002] What it does: It uses the smartphone's microphone to capture the audio during a call and streams it to a server as digital audio data, which is sent in packets.

[1003] Input: Call audio

[1004] Output: Streaming digital audio data

[1005] Step 3:

[1006] The server passes the voice data to the speech recognition module and converts it into text.

[1007] What it does: It passes the received digital audio data to the Google Cloud Speech-to-Text API, which converts the audio into text, which is then used in the next analysis step.

[1008] Input: Digital audio data

[1009] Output: Text data

[1010] Step 4:

[1011] The server uses a generative AI model to analyze the text data and detect fraudulent keywords and phrases.

[1012] How it works: Using a generative AI model (such as BERT or GPT-4) to detect specific keywords or phrases in text data, such as "It's me" or "I need money urgently."

[1013] Input: Text data

[1014] Output: Fraudulent keyword and phrase detection results

[1015] Step 5:

[1016] The server scores the likelihood of fraud based on detected keywords and phrases.

[1017] What it does: Based on the keywords and phrases detected, it applies an algorithm to assess the likelihood of fraud and calculates a fraud risk score. It then determines the risk of the call.

[1018] Input: Fraudulent keyword or phrase detection results

[1019] Output: Fraud risk score

[1020] Step 6:

[1021] The server records potentially fraudulent calls and stores them in secure storage.

[1022] What it does: If the fraud risk score is determined to be high, the system will begin recording the call and store the recording in secure cloud storage such as AWS S3. After recording is complete, the data is encrypted and stored.

[1023] Input: Fraud Risk Score

[1024] Output: Save call recording data

[1025] Step 7:

[1026] Sends a warning notification if the server is at high fraud risk

[1027] Specific behavior: If the fraud risk score is high, a warning message is sent to the user and pre-designated notification recipients. Notifications such as "Possible fraud" are sent via push notification, email, or SMS using notification APIs such as Twilio.

[1028] Input: Fraud Risk Score

[1029] Output: Sending a warning message

[1030] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1031] The present invention combines a system that analyzes call content in real time and responds quickly to potential fraudulent activity with an emotion engine that recognizes user emotions. The system monitors call content from the moment the user initiates a call and analyzes specific keywords, phrases, and even voice characteristics to more accurately determine the likelihood of fraud. If a fraudulent activity is suspected, the system records the call and sends a warning to designated notification recipients.

[1032] System Configuration

[1033] 1. Call detection

[1034] The server waits for and receives a call start signal from the terminal to detect when the user has started a call on the phone, and when the server receives this signal, it starts call monitoring in real time.

[1035] 2. Streaming of call content

[1036] The terminal streams the voice to the server in real time when the user starts a call. The terminal has the function of converting the voice during the call into digital data and sending this data to the server.

[1037] 3. Voice Recognition

[1038] The server passes the received voice data to a speech recognition module, which converts the voice into text using a pre-trained model.

[1039] 4. Text Analysis

[1040] The generative AI analyzes the text data sent from the voice recognition module to detect specific fraud patterns and keywords, while also analyzing voice characteristics using an emotion engine to recognize the user's emotions.

[1041] 5. Determining the likelihood of fraud

[1042] The generative AI and emotion engine combine the results of their analysis to determine the likelihood of fraud. The generative AI calculates the probability of fraud for each keyword or phrase, and then takes into account the results of the emotion engine's analysis of user sentiment to precisely assess the likelihood of fraud.

[1043] 6. Recording and saving of call content

[1044] The server records calls that are deemed likely to be fraudulent and stores the data in secure storage, which can then be used for later review or as evidence.

[1045] 7. Sending Notifications

[1046] When a potentially fraudulent call is detected, the server sends a warning to the user and designated contacts (family and relatives). Notification methods can be selected from push notifications, emails, SMS, etc.

[1047] A natural language description of the program's processing

[1048] Call detection

[1049] When a user starts a call, the server detects the call start event and prepares for real-time analysis. For example, when an elderly person starts a call on their home landline phone, the server automatically detects the call.

[1050] Streaming of call content

[1051] The device transmits the call audio to the server in real time. The device is equipped with a microphone that converts the audio into digital data and sends it to the server as packets.

[1052] Voice Recognition

[1053] The server passes the received voice data to a speech recognition module, which converts the voice data into text data that is used in further analysis steps.

[1054] Text and Sentiment Analysis

[1055] The generative AI analyzes the text data sent from the voice recognition module to detect specific keywords and phrases. At the same time, the emotion engine analyzes the voice characteristics to recognize the user's emotions. For example, the emotion engine analyzes the tone, speed, and volume of the user's voice to determine stress or urgency.

[1056] Determining the likelihood of fraud

[1057] Generative AI combines the results of analytics and sentiment analysis to assess the likelihood of fraud. This process involves scoring the fraud risk for each keyword or phrase, and then taking into account the results of the sentiment engine to make a final decision.

[1058] Call recording and storage

[1059] The server records any audio data that the AI ​​determines to be fraudulent and stores it in secure storage. For example, recording begins when a phrase such as "transfer your money now" is detected.

[1060] Sending notifications

[1061] The server will notify the user and the designated contacts that a potentially fraudulent call has been detected. The notification will be sent as a warning message saying, "Your mother's call may be fraudulent."

[1062] In this way, the present invention provides a system that analyzes call content in real time and combines it with an emotion engine to more accurately determine the possibility of fraud and respond quickly. Specifically, the system automatically starts working the moment the user starts a call, and if a possible fraud is detected, a notification is sent immediately, preventing fraud damage before it occurs.

[1063] The processing flow will be explained below.

[1064] Step 1:

[1065] The server detects when a user starts a call on the phone. The server receives a call start signal from the user's device and prepares for real-time call monitoring. Specifically, the server receives a signal from the device as soon as the call starts and starts the analysis module.

[1066] Step 2:

[1067] When a user initiates a call, the device streams the call audio to the server in real time. The device digitizes the audio picked up through the microphone and sends the data to the server. For example, audio data from a landline phone or smartphone is continuously sent to the server.

[1068] Step 3:

[1069] The server passes the received voice data to a speech recognition module, which converts it into text data. The speech recognition module uses a pre-trained model to analyze the voice and convert it into corresponding text, which is used in the next analysis step.

[1070] Step 4:

[1071] The generation AI receives the text data sent from the speech recognition module and begins analyzing it to detect specific fraud patterns and keywords. At the same time, the emotion engine analyzes this voice data to recognize the user's emotions. For example, the generation AI detects fraud-specific phrases such as "I need money urgently," and the emotion engine analyzes the user's tone and speed of voice to indicate feelings of tension or impatience.

[1072] Step 5:

[1073] The generative AI and emotion engine combine the analysis results to determine the likelihood of fraud. First, the generative AI calculates a fraud score for the detected keywords and phrases, and then the emotion engine makes a final assessment based on the user's emotion results identified by the generative AI. If it determines that there is a high likelihood of fraud, it returns the result to the server.

[1074] Step 6:

[1075] The server then issues instructions to record calls deemed likely to be fraudulent and store the data in secure storage, where it can later be used for verification or evidence. This step is triggered if specific phrases, such as "I need money now," are detected.

[1076] Step 7:

[1077] If the server detects a potentially fraudulent call, it will send a warning to the user and designated recipients (for example, family and relatives). Notification methods can be selected from push notifications, emails, SMS, etc. For example, a warning message stating, "There is a possibility that your mother's call may be fraudulent" will be sent.

[1078] Step 8:

[1079] The user and the designated contacts receive the warning from the server and check the details. The contacts, such as family members, check the warning message and, if necessary, contact the elderly person again to check the details of the call. At this time, they can also notify the police if necessary and take detailed measures.

[1080] In this way, the present invention analyzes the content of phone calls in real time and combines it with an emotion engine to create a system that can accurately determine the possibility of fraud and respond quickly, thereby preventing fraud damage before it occurs.

[1081] Example 2

[1082] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1083] As fraudulent phone calls increase, there is a problem that vulnerable people, such as the elderly, are particularly susceptible to these attacks. Conventional methods have not established a means to detect fraud during a call and respond quickly. Furthermore, there is a need for a highly accurate analysis method that takes into account the user's emotions.

[1084] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for detecting the start of a call, means for streaming the voice of the call to the central processing unit in real time, means for converting the voice to text, means for analyzing the text to determine the possibility of fraud and the user's emotions, means for recording the contents of the call if there is a high possibility of fraud, and means for sending a notification if there is a possibility of fraud. This makes it possible to accurately determine the possibility of fraud during a call and respond quickly.

[1085] The "means for detecting the start of a call" is a function that automatically recognizes the timing when the user starts a call and notifies the system of the start of the call.

[1086] "Means for streaming the voice of a call to a central processing unit in real time" refers to a function that converts the voice of a call into digital data and instantly transmits that data to a central processing unit.

[1087] The "means for converting voice to text" is a function that analyzes received voice data and converts it into text data.

[1088] The "means for analyzing text to determine the likelihood of fraud and the user's emotions" is a function that uses a generative language model and an emotion analysis device to evaluate the likelihood of fraud and the user's emotions from text data.

[1089] "Means for recording phone calls when there is a high risk of fraud" refers to a function that records the contents of phone calls as digital data when text analysis and sentiment analysis indicate a high risk of fraud.

[1090] "Means for sending notifications in the event of possible fraud" is a function that sends a warning message to the user and designated notification recipients when a risk of fraud is detected.

[1091] A "generative language model" is an algorithm that has been trained in advance on a large amount of language data, and is used to analyze input text data and detect specific patterns and keywords.

[1092] An "emotion analysis device" is a software or hardware device that recognizes a user's emotions from voice data or text data.

[1093] The present invention combines a system that analyzes call content in real time and responds quickly to potential fraud, with an emotion engine that recognizes user emotions. Specific embodiments of this system are described below.

[1094] The system mainly consists of the following elements:

[1095] server

[1096] Terminal

[1097] Voice Recognition Module

[1098] Generation AI

[1099] Emotion Engine

[1100] Secure Storage

[1101] Notification System

[1102] The server detects when a user starts a call by receiving a signal from the terminal. Upon receiving the call start signal, the server starts real-time analysis for call monitoring. When the terminal starts a call, it converts the voice data into digital data and streams it to the server in real time.

[1103] The speech recognition module converts the voice data received by the server into text data. This uses a generative AI model that has been trained on a large amount of data in advance. The generative AI model converts voice to text with high accuracy, removes background noise, and identifies the speaker.

[1104] The generative AI analyzes the text data sent from the speech recognition module to detect specific keywords and phrases. At the same time, the emotion engine analyzes voice characteristics (tone, speed, volume, etc.) to recognize the user's emotions. For example, if the user's voice sounds tense or urgent, the emotion engine will determine the level of stress or urgency.

[1105] Based on the results of this analysis, the generative AI performs a process to assess the likelihood of fraud. It scores the risk of fraud and integrates the results of the emotion engine to make a final judgment. For example, if a phrase such as "Please tell me the bank details" is detected and the tone of voice used is unstable, it will be assessed as having a high likelihood of fraud.

[1106] If the AI ​​generator determines that a call is likely to be fraudulent, the server records the call and stores the data in secure storage, which has access restrictions and can be accessed as needed.

[1107] Furthermore, when a potentially fraudulent call is detected, the server sends a warning to the user and designated contacts. Notification methods include push notification, email, and SMS. For example, a warning message stating, "Your mother's call may be fraudulent" is sent to the contact.

[1108] For example, if an elderly person utters the phrase "Please transfer the money now" during a call, the system will immediately detect this phrase and determine that it is likely a fraud. In this case, the server will record the call, store the data in secure storage, and send an alert to the designated notification recipient.

[1109] Examples of prompts include:

[1110] "Design a fraud detection and notification system when an elderly person utters a phrase that could be fraudulent during a call. The system should include a server that analyzes the call audio in real time, evaluates the possibility of fraud, and records the call and sends a notification if necessary."

[1111] In this way, by analyzing the content of phone calls in real time and combining it with an emotion engine, the present invention can accurately determine the possibility of fraud and respond quickly. The system operates automatically from the moment the user starts a call, and if a possible fraud is detected, a notification is sent immediately, preventing fraud damage before it occurs.

[1112] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1113] Processing Steps

[1114] Step 1:

[1115] The server detects that a user has started a call. It receives a call start signal from the terminal as input and processes this signal as a call start event. This causes the server to start preparing for call monitoring and prepare for real-time voice data analysis. Specifically, when the server receives the call start signal, it starts the analysis module.

[1116] Step 2:

[1117] The terminal converts the call voice into digital data and streams it to the server in real time. As input, the user's voice is collected through a microphone and the voice data is converted into digital data. As output, this digital data is divided into packets and sent to the server. Specifically, the voice conversion module of the terminal generates digital voice data, and the network module sends it to the server.

[1118] Step 3:

[1119] The server passes the received voice data to a voice recognition module, which converts the voice into text data. As input, the digital voice data transmitted in real time is passed to the voice recognition module. As output, this voice data is converted into text data, which is used for the next analysis step. Specifically, the voice recognition module analyzes the voice data using a generative AI model and generates text data.

[1120] Step 4:

[1121] The generation AI analyzes the text data sent from the voice recognition module to detect specific keywords and phrases. At the same time, the emotion engine analyzes the voice characteristics and recognizes the user's emotions. Text data and voice characteristics data are provided as input. The output is an assessment of the likelihood of fraud and the user's emotional state. Specifically, the generation AI checks for the presence of specified keywords and phrases, and the emotion engine analyzes the tone, speed, and volume of the voice.

[1122] Step 5:

[1123] Generative AI integrates the results of analysis and sentiment analysis to assess the likelihood of fraud. The inputs are the detection results of specific keywords and phrases and the analysis results of the sentiment engine. The output is a fraud risk score and a final judgment. Specifically, a risk score is calculated for each keyword or phrase, and the results of sentiment analysis are then added to make the final judgment.

[1124] Step 6:

[1125] If the generation AI determines that there is a high possibility of fraud, the server records the call and stores the data in secure storage. The input is call data with a high fraud risk assessment. The output is the recorded voice data, which is encrypted and stored. Specifically, the call is recorded in real time, and the data is sent to secure storage via an encryption module.

[1126] Step 7:

[1127] The server notifies the user and designated contacts that a potentially fraudulent call has been detected. Event information about the call that was determined to be at high risk of fraud is provided as input. A warning message is sent as output via push notification, email, SMS, etc. Specifically, a notification API is used to send a message to the user or family member stating, "There has been a potentially fraudulent call."

[1128] This is the flow of the program processing for this system. At each step, specific data processing and calculations are performed, allowing for rapid response in cases where there is a high possibility of fraud.

[1129] (Application example 2)

[1130] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1131] In recent years, telephone fraud has been on the rise, with elderly people being particularly vulnerable to its fallout. Because such fraudulent activities use sophisticated persuasive and psychological tactics, it is difficult to ensure sufficient detection accuracy by simply analyzing keywords and phrases in phone call content. Furthermore, even if a possible fraud is detected, it is difficult to prevent damage without a rapid response. Therefore, a system is needed that can accurately determine the possibility of fraud and respond quickly by taking into account the user's emotional state in addition to real-time analysis of phone call content.

[1132] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for detecting the start of a call, means for streaming the call audio to the server in real time, means for converting the audio into text, and means for analyzing the text to determine the possibility of fraud. This enables real-time analysis of the call content. The system also includes means for recording the call content when there is a high possibility of fraud, means for sending a notification when there is a possibility of fraud, and an emotion engine for analyzing the user's emotions in real time, and is equipped with means for precisely evaluating the possibility of fraud by integrating the generative AI model and the emotion engine. This improves the accuracy of determining the possibility of fraud and enables prompt prevention of damage.

[1133] The "means for detecting the start of a call" is a function for automatically detecting the moment when a user starts a call on the telephone.

[1134] "Means for streaming the voice of a call to a server in real time" refers to a function that digitizes the voice data during a call in real time and transmits it to a server via the Internet.

[1135] "Means for converting voice to text" refers to voice recognition technology that analyzes received voice data and converts it into text data.

[1136] "Means for analyzing text to determine the possibility of fraud" refers to a function that analyzes text data converted from speech and detects specific fraudulent keywords and phrases.

[1137] "Means for recording phone calls when there is a high possibility of fraud" refers to a function that records and saves as a digital audio file any phone call that is determined to be highly likely to be fraudulent as a result of analysis.

[1138] "Means for sending notifications in the event of possible fraud" is a function that sends a warning to the user and designated notification recipients when a call with a high risk of fraud is detected.

[1139] The "emotion engine that analyzes user emotions in real time" is a technology that analyzes voice characteristics (e.g., tone, speed, volume) during a call and recognizes the user's emotional state (e.g., stress, urgency) in real time.

[1140] A "generative AI model" is an artificial intelligence model that has been trained in advance with large amounts of data, and is a technology used to detect specific keywords and phrases with high accuracy.

[1141] "Means of precisely assessing the likelihood of fraud by integrating an emotion engine" is a function that integrates the text analysis results of the generative AI model with the emotion analysis results of the emotion engine to comprehensively determine the likelihood of fraud and improve its accuracy.

[1142] The "means for receiving notifications and allowing users to check the content of fraudulent calls" is an interface function that allows users to check the warning notifications they receive and view the details of calls suspected to be fraudulent.

[1143] System program generation

[1144] Hardware and Software Use

[1145] The system for implementing this invention mainly consists of a server and a user's smartphone. The server is equipped with a high-speed processor and a large-capacity SSD, and runs a speech recognition model (e.g., Google Speech-to-Text), a sentiment analysis engine (e.g., IBM Watson Tone Analyzer), and a generative AI model (e.g., OpenAI's GPT-4).

[1146] A smartphone uses the following hardware and software:

[1147] Microphone: Captures call audio.

[1148] Communications module: Streams call audio to the server in real time.

[1149] Internet connection: Allows for high speed transmission of data.

[1150] A natural language description of the program's processing

[1151] Detecting the start of a call

[1152] When a user starts a call on the phone, the smartphone application detects the call start event and notifies the server, which receives the notification and begins preparing for real-time analysis.

[1153] Streaming call audio

[1154] The user's smartphone picks up the call audio with a microphone, converts it into digital data, and sends this data over the Internet to a server, which receives the audio streaming data in real time.

[1155] Speech-to-text

[1156] The server converts the received voice data into text using a speech recognition model (e.g., Google Speech-to-Text), which is then used for subsequent analysis.

[1157] Text and Sentiment Analysis

[1158] A server-based generative AI model (e.g., GPT-4) analyzes the text data obtained from the speech recognition module to detect specific fraudulent keywords and phrases. At the same time, a sentiment analysis engine (e.g., IBM Watson Tone Analyzer) analyzes voice characteristics to recognize the user's emotions in real time. This process is carried out using advanced algorithms.

[1159] Determining the likelihood of fraud

[1160] The server combines the results of the generative AI model and the sentiment analysis engine to precisely assess the likelihood of fraud. It scores the fraud risk by combining specific keywords and the user's emotional state (e.g., high stress). If it determines there is a high likelihood of fraud, the server records the call and stores it in secure storage.

[1161] Warning notice

[1162] The server sends a warning to the user and pre-designated notification recipients (e.g., family members) for calls that are determined to be highly likely to be fraudulent. This notification can be done by push notification, email, SMS, or other means, and will send a message such as, "Signs of fraud have been detected in the content of your call."

[1163] Examples of concrete examples and prompts

[1164] When an elderly person receives a scam call pretending to be their grandchild, the emotion analysis engine detects that the tone and speed of the elderly person's voice warrants attention. The generative AI model picks up phrases such as "transfer money." This causes the server to assess the fraud risk as high, and recording begins. A notification is also sent to the family saying, "There is a possibility that your mother's call is a scam."

[1165] Prompt Sentence Examples

[1166] "Detect potential fraudulent elements in the following phone calls."

[1167] "Assess the user's stress level based on tone of voice and emotional analysis."

[1168] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1169] Step 1:

[1170] Detect the start of a call

[1171] Input: The user initiates a call.

[1172] The server receives a call start event from the smartphone application and detects the start of a call, which allows the server to prepare for real-time analysis.

[1173] Output: Call start information.

[1174] Specific behavior: When a user starts a call on their smartphone, the application catches the event and sends a call start signal to the server.

[1175] Step 2:

[1176] Streaming call audio

[1177] Input: Smartphone call audio.

[1178] The smartphone picks up the call audio through a microphone, converts it into digital data, and streams this data to a server over the internet.

[1179] Output: Real-time audio data.

[1180] How it works: The smartphone's microphone picks up sound, converts the sound signal into digital packets in real time, and sends them to a server. Because this data is sent in real time, it can be analyzed without delay.

[1181] Step 3:

[1182] Speech-to-text

[1183] Input: Real-time audio data.

[1184] The server sends the received voice data to a speech recognition model (e.g., Google Speech-to-Text) and converts the voice into text.

[1185] Output: Text data.

[1186] How it works: A server-based speech recognition model analyzes the audio data and converts it into corresponding text data, which is then used in subsequent analysis steps.

[1187] Step 4:

[1188] Text and Sentiment Analysis

[1189] Input: Text data of speech recognition results.

[1190] A generative AI model (e.g., GPT-4) analyzes text data to detect specific fraudulent keywords and phrases, while an emotion engine (e.g., IBM Watson Tone Analyzer) analyzes voice characteristics to assess user sentiment in real time.

[1191] Output: Deception probability and emotional state assessment results.

[1192] How it works: A generative AI model examines text to identify keywords and phrases related to fraud, while an emotion engine analyzes the tone, rate, and volume of speech to identify emotional states.

[1193] Step 5:

[1194] Determining the likelihood of fraud

[1195] Input: Deception likelihood and emotional state assessment results.

[1196] The server combines the results of the generative AI model and the emotion engine to assess the likelihood of fraud, and if the fraud risk is determined to be high, the call is recorded.

[1197] Output: Fraud risk score and audio recording.

[1198] How it works: The server combines the results of text analysis and sentiment analysis to score the risk of fraud. If the score exceeds a threshold, the call is recorded and stored in a secure location.

[1199] Step 6:

[1200] Warning notice

[1201] Input: Fraud risk score and audio recording.

[1202] The server will send a warning to the user and designated contacts if fraud is likely.

[1203] Output: A warning message.

[1204] What happens: The server uses the notification system to send a warning message to the user and their family members via push notification, email, SMS, etc., such as "Possible fraud detected."

[1205] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1206] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1207] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1208] [Fourth embodiment]

[1209] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1210] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1211] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1212] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1213] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1214] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1215] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1216] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1217] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1218] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1219] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1220] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1221] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1222] This invention is a system that analyzes call content in real time and responds quickly to potential fraud. The system monitors call content when a user initiates a call and uses generative AI to detect specific keywords and phrases to determine the possibility of fraud. If fraud is likely, the system records the call and sends a warning to designated notification recipients.

[1223] System Configuration

[1224] 1. Call detection

[1225] The server detects when a user initiates a call on the telephone by receiving a call initiation signal from the user terminal.

[1226] 2. Streaming of call content

[1227] The device streams the call audio to the server in real time. The device has a microphone that converts the audio into digital data and transmits the data to the server.

[1228] 3. Voice Recognition

[1229] The server passes the received voice data to a speech recognition module that converts the voice data to text, which uses a pre-trained model to convert speech to text with high accuracy.

[1230] 4. Text Analysis

[1231] Generative AI analyzes the converted call content to detect specific fraud patterns and keywords, using algorithms to assess the likelihood of fraud.

[1232] 5. Determining the likelihood of fraud

[1233] The generation AI determines whether there is a possibility of fraud based on the analysis results, and if there is a high possibility of fraud, it returns the result to the server.

[1234] 6. Recording and saving of call content

[1235] If the generation AI determines that there is a high possibility of fraud, the server will record the call and store it in secure data storage.

[1236] 7. Sending Notifications

[1237] If the server detects a potentially fraudulent call, it will send a warning to the user and their designated contacts via push notification, email, SMS, or other methods.

[1238] A natural language description of the program's processing

[1239] Call detection

[1240] When a user starts a call, the server detects the call start event and prepares for real-time analysis. For example, when an elderly person starts a call on their home landline phone, the server automatically detects the call.

[1241] Streaming of call content

[1242] The terminal transmits the voice during the call to the server in real time. The actual voice is converted into digital data and sent as packets to the server.

[1243] Voice Recognition

[1244] The server analyzes the received voice data and converts it into text using a speech recognition module, which is then used in the next analysis step.

[1245] Text analytics

[1246] The generation AI analyzes the text data received from the voice recognition module and detects specific keywords and phrases to determine whether the data is fraudulent. For example, if the data contains phrases such as "It's me" or "I need money urgently," the generation AI will detect this.

[1247] Determining the likelihood of fraud

[1248] The generative AI evaluates the likelihood of fraud based on the analysis results and returns the score to the server, flagging cases where fraud is highly likely.

[1249] Call recording and storage

[1250] The server records calls that are deemed likely to be fraudulent and stores the data in secure storage. For example, recording begins when a phrase such as "I want to transfer money" is detected.

[1251] Sending notifications

[1252] The server notifies the user and the designated contacts that a potentially fraudulent call has been detected. The notification may be sent as a warning message such as, "Your mother's call may be fraudulent."

[1253] In this way, the present invention aims to prevent fraud by analyzing the contents of calls in real time, and if there is a possibility of fraud, quickly recording the contents of the call and making appropriate notifications, thereby preventing fraud damage from occurring.

[1254] The processing flow will be explained below.

[1255] Step 1:

[1256] The server waits for and receives a call start signal from the terminal to detect when the user has started a call on the phone, and when the server receives this signal, it starts monitoring the call in real time.

[1257] Step 2:

[1258] The device streams the audio to the server in real time as soon as the user starts a call. The device has a microphone that converts the audio into digital data and sends it to the server.

[1259] Step 3:

[1260] The server passes the received voice data to a speech recognition module, which analyzes the voice data and converts it into corresponding text data in real time, ready for the subsequent analysis step.

[1261] Step 4:

[1262] The Generative AI receives the text data sent from the speech recognition module and begins analyzing it to detect specific fraud patterns and keywords. Using pre-trained models, the Generative AI scores the likelihood of fraud.

[1263] Step 5:

[1264] The generation AI determines whether there is a possibility of fraud based on the analysis results. The generation AI calculates the probability of fraud for each keyword or phrase and returns the result to the server as a score. If there is a high possibility of fraud, it issues instructions to the server.

[1265] Step 6:

[1266] If the server determines that a call is likely to be fraudulent, it automatically records the call audio and stores the recorded audio in secure data storage for later review.

[1267] Step 7:

[1268] When a potentially fraudulent call is detected, the server sends a warning to the user and designated contacts (family and relatives) via push notification, email, SMS, etc.

[1269] Step 8:

[1270] The user and the designated contacts will receive a warning from the server. If the user or their family members determine that the information is likely fraudulent, they can take appropriate action, such as further investigating the matter or reporting it to the police.

[1271] For example, when an elderly person starts a call on their home landline, the server automatically detects the call and analyzes the content in real time. If a phrase such as "I need money urgently" is detected during the call, the server records the call and simultaneously sends a warning to the family. By receiving this warning, the family can prevent fraud before it occurs.

[1272] Example 1

[1273] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1274] In recent years, telephone fraud has been on the rise, with many victims reported, particularly among the elderly. To prevent such fraud, a system is needed that can monitor call content in real time and quickly detect possible fraud. However, current systems have difficulty analyzing call content in real time and accurately determining the possibility of fraud. Therefore, there is a need for a method that can accurately detect possible fraud during a call and respond quickly.

[1275] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1276] In this invention, the server includes means for detecting the start of a call, means for streaming the call audio to the server in real time, means for converting the audio into text, means for analyzing the text using a model for detecting specific keywords and phrases to determine the possibility of fraud, means for recording the content of the call and saving it in secure data storage if there is a high possibility of fraud, means for sending a notification if there is a possibility of fraud, and means for receiving the notification and allowing the user to confirm the content of the fraudulent call. This makes it possible to detect the possibility of fraud during a call with high accuracy in real time and take necessary measures quickly.

[1277] A "means for detecting the start of a call" is a device or system for detecting the moment when a user starts a telephone call.

[1278] "Means for streaming telephone call audio to a server in real time" refers to a device or system for converting telephone call audio into digital data in real time and transmitting it to a server via the Internet.

[1279] A "means for converting speech to text" is a device or system that uses speech recognition technology to analyze received speech data and convert it into text data.

[1280] A "model for detecting specific keywords and phrases" is a trained algorithm used to analyze text data during calls and detect specific fraud-related keywords and phrases.

[1281] The "means for determining the likelihood of fraud" is a device or system that evaluates the likelihood of fraud based on the analyzed text data and provides the results.

[1282] A "call recording device" is a device or system that records calls that are determined to be likely fraudulent and stores the data.

[1283] "Secure Data Storage" means an encrypted data storage device or system for securely storing recordings.

[1284] A "means for sending a notification" is a device or system for sending an alert to designated notification contacts when a potentially fraudulent call is detected.

[1285] The "means for receiving the notification and allowing the user to check the content of the fraudulent call" refers to a device or system that receives the sent notification and allows the user to check the content of the notification.

[1286] This invention is a system that analyzes call content in real time and responds quickly to potential fraud. The system monitors call content as soon as the user initiates a call and uses a generative AI model to detect specific keywords and phrases to determine the likelihood of fraud. If fraud is likely, the system records the call and sends a warning to designated contacts.

[1287] This system is implemented using the following hardware and software:

[1288] Hardware: Servers, user devices (smartphones, landlines, etc.)

[1289] Software: speech recognition modules (e.g., Google Cloud Speech-to-Text), generative AI models (e.g., OpenAI GPT-4)

[1290] Call detection

[1291] When a user starts a call on the phone, the server receives a call start signal from the user terminal and detects the start of the call. For example, when an elderly person starts a call on their home landline phone, the server receives the call start signal emitted from the terminal and confirms that the call has started.

[1292] Streaming of call content

[1293] After a call starts, the device streams the call audio to the server in real time. Specifically, the device's built-in microphone converts the audio into digital data and sends that data to the server via the Internet. For example, a smartphone's microphone picks up the audio, digitizes it, and sends it to the server in real time as packets.

[1294] Voice Recognition

[1295] The server passes the received voice data to a speech recognition module (e.g., Google Cloud Speech-to-Text) and converts the voice into text. For example, if a user says "hello" during a phone conversation, the word will be converted into text as "Konnichiwa."

[1296] Text analytics

[1297] The generation AI analyzes the text data sent from the voice recognition module and detects specific keywords and phrases (e.g., "It's me," "I need money urgently," etc.). For example, if the call text contains the phrase "Please transfer the money immediately," the generation AI will detect this and calculate a score indicating the likelihood of fraud.

[1298] Determining the likelihood of fraud

[1299] The generative AI evaluates the likelihood of fraud based on the analysis results and returns the result to the server. For example, if a specific fraud-related phrase is detected, a high fraud risk score is set and the score is sent to the server.

[1300] Call recording and storage

[1301] The server records calls that are deemed likely to be fraudulent and stores the data in secure data storage. For example, recording begins the moment the user responds to the fraudster by saying, "I'll transfer the money." The recording data is then securely stored in encrypted storage.

[1302] Sending notifications

[1303] The server notifies the user and designated contacts that a potentially fraudulent call has been detected. Notifications can be sent in the form of push notifications, emails, SMS, etc. For example, a warning message stating "Your mother's call may be fraudulent" could be sent to the user's family.

[1304] As described above, the present invention aims to prevent fraud by analyzing the contents of calls in real time, and if there is a possibility of fraud, quickly recording the contents of the call and making appropriate notifications, thereby preventing fraud damage from occurring.

[1305] Prompt Sentence Examples

[1306] Examples of prompts to input to a generative AI model include:

[1307] Please explain in natural language the programming steps of the generative AI that analyzes user conversations in real time and detects specific keywords and phrases. Please also provide specific examples.

[1308] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1309] System program processing flow

[1310] Step 1: Call detection

[1311] Input: Signal to start a call

[1312] Processing: The server receives a call initiation signal from the user terminal to detect that the user is about to initiate a call.

[1313] Specific operation: For example, when an elderly person starts a call using their home landline, the device detects this and sends a signal to the server, which then receives the signal and confirms that the call has started.

[1314] Output: Call start confirmation information

[1315] Step 2: Stream the call

[1316] Input: Call audio data

[1317] Processing: The device streams the call audio to the server in real time, converts the call audio into digital data, and sends the data over the internet to the server.

[1318] How it works: For example, a smartphone's microphone picks up the audio of a call and converts it into digital data, which is then split into packets and streamed to a server.

[1319] Output: Digitized voice data

[1320] Step 3: Voice Recognition

[1321] Input: Digitized voice data

[1322] Processing: The server passes the received voice data to a speech recognition module (e.g., Google Cloud Speech-to-Text) to convert the voice into text.

[1323] Specific operation: For example, if the voice data contains "Hello", it will be converted into text data as "Konnichiwa".

[1324] Output: Text data

[1325] Step 4: Text Analysis

[1326] Input: Text data

[1327] Processing: The generative AI analyzes the text data sent from the speech recognition module to detect specific fraud-related keywords and phrases (e.g., "It's me," "I need money urgently," etc.).

[1328] Specific behavior: For example, if the text data contains the phrase "Please transfer the money immediately," the generative AI will detect this and calculate a score indicating the likelihood of fraud.

[1329] Output: Keyword analysis results and fraud likelihood score

[1330] Step 5: Determine the likelihood of fraud

[1331] Input: Keyword analysis results and fraud likelihood score

[1332] Processing: The generative AI assesses the likelihood of fraud based on the analyzed text data and score.

[1333] Specific behavior: For example, if the fraud likelihood score is set high, the generation AI will return that score to the server.

[1334] Output: Fraud likelihood assessment result

[1335] Step 6: Record and save the call

[1336] Input: Fraud likelihood assessment results

[1337] Processing: The server records calls that are determined to be likely fraudulent and stores the data in secure storage.

[1338] Specific operation: For example, the moment the user responds to the scammer by saying "I'll transfer the money," recording begins, and the recording data is stored in encrypted storage.

[1339] Output: Recorded and securely stored data

[1340] Step 7: Sending notifications

[1341] Input: Fraud likelihood assessment results and audio recording

[1342] Action: The server notifies the user and designated contacts that a potentially fraudulent call has been detected.

[1343] Specific behavior: Notifications can be sent in the form of push notifications, emails, SMS, etc. For example, a warning message stating "Your mother's call may be fraudulent" is sent to the user's family.

[1344] Output: Information message

[1345] Through these steps, the system can analyze call content in real time, quickly detect potential fraud, and take necessary action.

[1346] (Application example 1)

[1347] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1348] In modern society, telephone fraud continues to affect many people. Elderly and other vulnerable groups are particularly susceptible to fraud, easily falling prey to the sophisticated tactics of criminals. Conventional methods have made it difficult to monitor phone conversations in real time, quickly identify potential fraud, and respond accordingly. There is also a need for a system that can quickly notify a third party of potentially fraudulent calls and take appropriate action. Given this background, there is an urgent need to develop a system that can analyze potential fraud in real time during a call and take appropriate action.

[1349] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1350] In this invention, the server includes means for detecting the start of a call, means for streaming the call audio to the server in real time, means for converting the audio to text, means for analyzing the text to determine the possibility of fraud, means for recording the call if fraud is likely, means for sending a notification if fraud is likely, means for including a generative AI model for analyzing the text of the call, and means for sending a notification of suspected fraud to a notification recipient if fraud is likely. This enables fraudulent activity during a call to be detected in real time and appropriate action to be taken promptly.

[1351] The "means for detecting the start of a call" is a device or software that has the function of recognizing and notifying that a call has started.

[1352] "Means for streaming the audio of a call to a server in real time" refers to a device or software that has the function of converting audio data during a call into a digital format in real time and transmitting it to a server.

[1353] A "means for converting speech to text" is a device or software that has the function of analyzing speech data and converting it into corresponding text data.

[1354] A "means for analyzing text to determine likelihood of fraud" is a device or software capable of examining text data to identify fraud-related keywords and phrases and assess the likelihood of fraud.

[1355] "Means for recording telephone conversations when fraud is likely" refers to devices or software that have the function of automatically recording telephone conversations when fraud is deemed to be likely.

[1356] A "means for sending notifications in the event of potential fraud" is a device or software capable of sending a warning message or notification to a specified recipient in the event of potential fraud.

[1357] A "means including a generative AI model for analyzing the text of a call" is a device or software that uses generative AI to convert voice data into text and analyzes that text to detect specific keywords or phrases.

[1358] "Means for sending a notice of suspected fraud to a notification recipient when there is a high possibility of fraud" refers to a device or software that has the function of notifying a pre-determined notification recipient of suspected fraud when it is determined that there is a high possibility of fraud.

[1359] The present invention provides a system that analyzes the contents of calls in real time and responds quickly when there is a possibility of fraud. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The following describes in detail the preferred embodiments of the present invention.

[1360] System Configuration

[1361] The system is realized using the user's smartphone, a server, and a generative AI model.

[1362] Hardware and software used

[1363] Hardware: Smartphones, servers

[1364] Software: Android / iOS SDK, Google Cloud Speech-to-Text API, Fraud Detection AI Model

[1365] System Operation

[1366] 1. Call detection:

[1367] The user's smartphone detects the start of a call, which is notified using the phone application's integration API.

[1368] 2. Streaming Call Content:

[1369] During a call, the smartphone streams the audio in real time to a server, using the smartphone's built-in microphone to convert the audio data into a digital format and send it to the server.

[1370] 3. Speech Recognition:

[1371] The server converts the received voice data into text using the Google Cloud Speech-to-Text API.

[1372] 4. Text Analysis:

[1373] The server uses a generative AI model (e.g., based on BERT or GPT-4) to analyze the text data and detect keywords and phrases related to fraud.

[1374] 5. Determining the likelihood of fraud:

[1375] The server scores the likelihood of fraud based on the keywords and phrases detected.

[1376] 6. Recording and storage of calls:

[1377] If the server determines that there is a high possibility of fraud, it will automatically record the call and store it in secure storage (e.g., Amazon S3).

[1378] 7. Sending Notifications:

[1379] If there is a high possibility of fraud, the server will send a warning message or a notification to the user and pre-specified notification destinations using a notification API such as Twilio.

[1380] Specific examples

[1381] For example, if an elderly parent receives a call from a scammer, here's how it works:

[1382] 1. An elderly parent's smartphone detects the start of a call.

[1383] 2. The call content is sent to the server in real time.

[1384] 3. The server converts the speech into text through speech recognition.

[1385] 4. The generative AI model detects fraudulent keywords such as "It's me" and "I need money."

[1386] 5. If a call is deemed likely to be fraudulent, the call will be recorded and a warning will be sent to the user and their family.

[1387] 6. A warning message will be sent, for example, a notification saying "Your mother's call may be fraudulent."

[1388] Prompt Sentence Examples

[1389] "I want to create an app to protect against fraud. The app will analyze phone calls in real time, detect specific fraudulent keywords and phrases, and issue a warning. The prompt might look like this:

[1390] Please analyze this call to see if it contains any scam phrases, and if so, please let us know what they are."

[1391] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1392] Step 1:

[1393] The smartphone detects the start of a call

[1394] Specific operation: When a user starts a call using the phone app, a call start event occurs, and the start of the call is notified to the server via the smartphone's collaboration API. This event is sent to the server as call log information.

[1395] Input: Signal to start a call

[1396] Output: Call start notification and call log information

[1397] Step 2:

[1398] The smartphone streams the call content to the server in real time.

[1399] What it does: It uses the smartphone's microphone to capture the audio during a call and streams it to a server as digital audio data, which is sent in packets.

[1400] Input: Call audio

[1401] Output: Streaming digital audio data

[1402] Step 3:

[1403] The server passes the voice data to the speech recognition module and converts it into text.

[1404] What it does: It passes the received digital audio data to the Google Cloud Speech-to-Text API, which converts the audio into text, which is then used in the next analysis step.

[1405] Input: Digital audio data

[1406] Output: Text data

[1407] Step 4:

[1408] The server uses a generative AI model to analyze the text data and detect fraudulent keywords and phrases.

[1409] How it works: Using a generative AI model (such as BERT or GPT-4) to detect specific keywords or phrases in text data, such as "It's me" or "I need money urgently."

[1410] Input: Text data

[1411] Output: Fraudulent keyword and phrase detection results

[1412] Step 5:

[1413] The server scores the likelihood of fraud based on detected keywords and phrases.

[1414] What it does: Based on the keywords and phrases detected, it applies an algorithm to assess the likelihood of fraud and calculates a fraud risk score. It then determines the risk of the call.

[1415] Input: Fraudulent keyword or phrase detection results

[1416] Output: Fraud risk score

[1417] Step 6:

[1418] The server records potentially fraudulent calls and stores them in secure storage.

[1419] What it does: If the fraud risk score is determined to be high, the system will begin recording the call and store the recording in secure cloud storage such as AWS S3. After recording is complete, the data is encrypted and stored.

[1420] Input: Fraud Risk Score

[1421] Output: Save call recording data

[1422] Step 7:

[1423] Sends a warning notification if the server is at high fraud risk

[1424] Specific behavior: If the fraud risk score is high, a warning message is sent to the user and pre-designated notification recipients. Notifications such as "Possible fraud" are sent via push notification, email, or SMS using notification APIs such as Twilio.

[1425] Input: Fraud Risk Score

[1426] Output: Sending a warning message

[1427] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1428] The present invention combines a system that analyzes call content in real time and responds quickly to potential fraudulent activity with an emotion engine that recognizes user emotions. The system monitors call content from the moment the user initiates a call and analyzes specific keywords, phrases, and even voice characteristics to more accurately determine the likelihood of fraud. If a fraudulent activity is suspected, the system records the call and sends a warning to designated notification recipients.

[1429] System Configuration

[1430] 1. Call detection

[1431] The server waits for and receives a call start signal from the terminal to detect when the user has started a call on the phone, and when the server receives this signal, it starts call monitoring in real time.

[1432] 2. Streaming of call content

[1433] The terminal streams the voice to the server in real time when the user starts a call. The terminal has the function of converting the voice during the call into digital data and sending this data to the server.

[1434] 3. Voice Recognition

[1435] The server passes the received voice data to a speech recognition module, which converts the voice into text using a pre-trained model.

[1436] 4. Text Analysis

[1437] The generative AI analyzes the text data sent from the voice recognition module to detect specific fraud patterns and keywords, while also analyzing voice characteristics using an emotion engine to recognize the user's emotions.

[1438] 5. Determining the likelihood of fraud

[1439] The generative AI and emotion engine combine the results of their analysis to determine the likelihood of fraud. The generative AI calculates the probability of fraud for each keyword or phrase, and then takes into account the results of the emotion engine's analysis of user sentiment to precisely assess the likelihood of fraud.

[1440] 6. Recording and saving of call content

[1441] The server records calls that are deemed likely to be fraudulent and stores the data in secure storage, which can then be used for later review or as evidence.

[1442] 7. Sending Notifications

[1443] When a potentially fraudulent call is detected, the server sends a warning to the user and designated contacts (family and relatives). Notification methods can be selected from push notifications, emails, SMS, etc.

[1444] A natural language description of the program's processing

[1445] Call detection

[1446] When a user starts a call, the server detects the call start event and prepares for real-time analysis. For example, when an elderly person starts a call on their home landline phone, the server automatically detects the call.

[1447] Streaming of call content

[1448] The device transmits the call audio to the server in real time. The device is equipped with a microphone that converts the audio into digital data and sends it to the server as packets.

[1449] Voice Recognition

[1450] The server passes the received voice data to a speech recognition module, which converts the voice data into text data that is used in further analysis steps.

[1451] Text and Sentiment Analysis

[1452] The generative AI analyzes the text data sent from the voice recognition module to detect specific keywords and phrases. At the same time, the emotion engine analyzes the voice characteristics to recognize the user's emotions. For example, the emotion engine analyzes the tone, speed, and volume of the user's voice to determine stress or urgency.

[1453] Determining the likelihood of fraud

[1454] Generative AI combines the results of analytics and sentiment analysis to assess the likelihood of fraud. This process involves scoring the fraud risk for each keyword or phrase, and then taking into account the results of the sentiment engine to make a final decision.

[1455] Call recording and storage

[1456] The server records any audio data that the AI ​​determines to be fraudulent and stores it in secure storage. For example, recording begins when a phrase such as "transfer your money now" is detected.

[1457] Sending notifications

[1458] The server will notify the user and the designated contacts that a potentially fraudulent call has been detected. The notification will be sent as a warning message saying, "Your mother's call may be fraudulent."

[1459] In this way, the present invention provides a system that analyzes call content in real time and combines it with an emotion engine to more accurately determine the possibility of fraud and respond quickly. Specifically, the system automatically starts working the moment the user starts a call, and if a possible fraud is detected, a notification is sent immediately, preventing fraud damage before it occurs.

[1460] The processing flow will be explained below.

[1461] Step 1:

[1462] The server detects when a user starts a call on the phone. The server receives a call start signal from the user's device and prepares for real-time call monitoring. Specifically, the server receives a signal from the device as soon as the call starts and starts the analysis module.

[1463] Step 2:

[1464] When a user initiates a call, the device streams the call audio to the server in real time. The device digitizes the audio picked up through the microphone and sends the data to the server. For example, audio data from a landline phone or smartphone is continuously sent to the server.

[1465] Step 3:

[1466] The server passes the received voice data to a speech recognition module, which converts it into text data. The speech recognition module uses a pre-trained model to analyze the voice and convert it into corresponding text, which is used in the next analysis step.

[1467] Step 4:

[1468] The generation AI receives the text data sent from the speech recognition module and begins analyzing it to detect specific fraud patterns and keywords. At the same time, the emotion engine analyzes this voice data to recognize the user's emotions. For example, the generation AI detects fraud-specific phrases such as "I need money urgently," and the emotion engine analyzes the user's tone and speed of voice to indicate feelings of tension or impatience.

[1469] Step 5:

[1470] The generative AI and emotion engine combine the analysis results to determine the likelihood of fraud. First, the generative AI calculates a fraud score for the detected keywords and phrases, and then the emotion engine makes a final assessment based on the user's emotion results identified by the generative AI. If it determines that there is a high likelihood of fraud, it returns the result to the server.

[1471] Step 6:

[1472] The server then issues instructions to record calls deemed likely to be fraudulent and store the data in secure storage, where it can later be used for verification or evidence. This step is triggered if specific phrases, such as "I need money now," are detected.

[1473] Step 7:

[1474] If the server detects a potentially fraudulent call, it will send a warning to the user and designated recipients (for example, family and relatives). Notification methods can be selected from push notifications, emails, SMS, etc. For example, a warning message stating, "There is a possibility that your mother's call may be fraudulent" will be sent.

[1475] Step 8:

[1476] The user and the designated contacts receive the warning from the server and check the details. The contacts, such as family members, check the warning message and, if necessary, contact the elderly person again to check the details of the call. At this time, they can also notify the police if necessary and take detailed measures.

[1477] In this way, the present invention analyzes the content of phone calls in real time and combines it with an emotion engine to create a system that can accurately determine the possibility of fraud and respond quickly, thereby preventing fraud damage before it occurs.

[1478] Example 2

[1479] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1480] As fraudulent phone calls increase, there is a problem that vulnerable people, such as the elderly, are particularly susceptible to these attacks. Conventional methods have not established a means to detect fraud during a call and respond quickly. Furthermore, there is a need for a highly accurate analysis method that takes into account the user's emotions.

[1481] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes means for detecting the start of a call, means for streaming the voice of the call to the central processing unit in real time, means for converting the voice to text, means for analyzing the text to determine the possibility of fraud and the user's emotions, means for recording the contents of the call if there is a high possibility of fraud, and means for sending a notification if there is a possibility of fraud. This makes it possible to accurately determine the possibility of fraud during a call and respond quickly.

[1482] The "means for detecting the start of a call" is a function that automatically recognizes the timing when the user starts a call and notifies the system of the start of the call.

[1483] "Means for streaming the voice of a call to a central processing unit in real time" refers to a function that converts the voice of a call into digital data and instantly transmits that data to a central processing unit.

[1484] The "means for converting voice to text" is a function that analyzes received voice data and converts it into text data.

[1485] The "means for analyzing text to determine the likelihood of fraud and the user's emotions" is a function that uses a generative language model and an emotion analysis device to evaluate the likelihood of fraud and the user's emotions from text data.

[1486] "Means for recording phone calls when there is a high risk of fraud" refers to a function that records the contents of phone calls as digital data when text analysis and sentiment analysis indicate a high risk of fraud.

[1487] "Means for sending notifications in the event of possible fraud" is a function that sends a warning message to the user and designated notification recipients when a risk of fraud is detected.

[1488] A "generative language model" is an algorithm that has been trained in advance on a large amount of language data, and is used to analyze input text data and detect specific patterns and keywords.

[1489] An "emotion analysis device" is a software or hardware device that recognizes a user's emotions from voice data or text data.

[1490] The present invention combines a system that analyzes call content in real time and responds quickly to potential fraud, with an emotion engine that recognizes user emotions. Specific embodiments of this system are described below.

[1491] The system mainly consists of the following elements:

[1492] server

[1493] Terminal

[1494] Voice Recognition Module

[1495] Generation AI

[1496] Emotion Engine

[1497] Secure Storage

[1498] Notification System

[1499] The server detects when a user starts a call by receiving a signal from the terminal. Upon receiving the call start signal, the server starts real-time analysis for call monitoring. When the terminal starts a call, it converts the voice data into digital data and streams it to the server in real time.

[1500] The speech recognition module converts the voice data received by the server into text data. This uses a generative AI model that has been trained on a large amount of data in advance. The generative AI model converts voice to text with high accuracy, removes background noise, and identifies the speaker.

[1501] The generative AI analyzes the text data sent from the speech recognition module to detect specific keywords and phrases. At the same time, the emotion engine analyzes voice characteristics (tone, speed, volume, etc.) to recognize the user's emotions. For example, if the user's voice sounds tense or urgent, the emotion engine will determine the level of stress or urgency.

[1502] Based on the results of this analysis, the generative AI performs a process to assess the likelihood of fraud. It scores the risk of fraud and integrates the results of the emotion engine to make a final judgment. For example, if a phrase such as "Please tell me the bank details" is detected and the tone of voice used is unstable, it will be assessed as having a high likelihood of fraud.

[1503] If the AI ​​generator determines that a call is likely to be fraudulent, the server records the call and stores the data in secure storage, which has access restrictions and can be accessed as needed.

[1504] Furthermore, when a potentially fraudulent call is detected, the server sends a warning to the user and designated contacts. Notification methods include push notification, email, and SMS. For example, a warning message stating, "Your mother's call may be fraudulent" is sent to the contact.

[1505] For example, if an elderly person utters the phrase "Please transfer the money now" during a call, the system will immediately detect this phrase and determine that it is likely a fraud. In this case, the server will record the call, store the data in secure storage, and send an alert to the designated notification recipient.

[1506] Examples of prompts include:

[1507] "Design a fraud detection and notification system when an elderly person utters a phrase that could be fraudulent during a call. The system should include a server that analyzes the call audio in real time, evaluates the possibility of fraud, and records the call and sends a notification if necessary."

[1508] In this way, by analyzing the content of phone calls in real time and combining it with an emotion engine, the present invention can accurately determine the possibility of fraud and respond quickly. The system operates automatically from the moment the user starts a call, and if a possible fraud is detected, a notification is sent immediately, preventing fraud damage before it occurs.

[1509] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1510] Processing Steps

[1511] Step 1:

[1512] The server detects that a user has started a call. It receives a call start signal from the terminal as input and processes this signal as a call start event. This causes the server to start preparing for call monitoring and prepare for real-time voice data analysis. Specifically, when the server receives the call start signal, it starts the analysis module.

[1513] Step 2:

[1514] The terminal converts the call voice into digital data and streams it to the server in real time. As input, the user's voice is collected through a microphone and the voice data is converted into digital data. As output, this digital data is divided into packets and sent to the server. Specifically, the voice conversion module of the terminal generates digital voice data, and the network module sends it to the server.

[1515] Step 3:

[1516] The server passes the received voice data to a voice recognition module, which converts the voice into text data. As input, the digital voice data transmitted in real time is passed to the voice recognition module. As output, this voice data is converted into text data, which is used for the next analysis step. Specifically, the voice recognition module analyzes the voice data using a generative AI model and generates text data.

[1517] Step 4:

[1518] The generation AI analyzes the text data sent from the voice recognition module to detect specific keywords and phrases. At the same time, the emotion engine analyzes the voice characteristics and recognizes the user's emotions. Text data and voice characteristics data are provided as input. The output is an assessment of the likelihood of fraud and the user's emotional state. Specifically, the generation AI checks for the presence of specified keywords and phrases, and the emotion engine analyzes the tone, speed, and volume of the voice.

[1519] Step 5:

[1520] Generative AI integrates the results of analysis and sentiment analysis to assess the likelihood of fraud. The inputs are the detection results of specific keywords and phrases and the analysis results of the sentiment engine. The output is a fraud risk score and a final judgment. Specifically, a risk score is calculated for each keyword or phrase, and the results of sentiment analysis are then added to make the final judgment.

[1521] Step 6:

[1522] If the generation AI determines that there is a high possibility of fraud, the server records the call and stores the data in secure storage. The input is call data with a high fraud risk assessment. The output is the recorded voice data, which is encrypted and stored. Specifically, the call is recorded in real time, and the data is sent to secure storage via an encryption module.

[1523] Step 7:

[1524] The server notifies the user and designated contacts that a potentially fraudulent call has been detected. Event information about the call that was determined to be at high risk of fraud is provided as input. A warning message is sent as output via push notification, email, SMS, etc. Specifically, a notification API is used to send a message to the user or family member stating, "There has been a potentially fraudulent call."

[1525] This is the flow of the program processing for this system. At each step, specific data processing and calculations are performed, allowing for rapid response in cases where there is a high possibility of fraud.

[1526] (Application example 2)

[1527] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1528] In recent years, telephone fraud has been on the rise, with elderly people being particularly vulnerable to its fallout. Because such fraudulent activities use sophisticated persuasive and psychological tactics, it is difficult to ensure sufficient detection accuracy by simply analyzing keywords and phrases in phone call content. Furthermore, even if a possible fraud is detected, it is difficult to prevent damage without a rapid response. Therefore, a system is needed that can accurately determine the possibility of fraud and respond quickly by taking into account the user's emotional state in addition to real-time analysis of phone call content.

[1529] The identification processing by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for detecting the start of a call, means for streaming the call audio to the server in real time, means for converting the audio into text, and means for analyzing the text to determine the possibility of fraud. This enables real-time analysis of the call content. The system also includes means for recording the call content when there is a high possibility of fraud, means for sending a notification when there is a possibility of fraud, and an emotion engine for analyzing the user's emotions in real time, and is equipped with means for precisely evaluating the possibility of fraud by integrating the generative AI model and the emotion engine. This improves the accuracy of determining the possibility of fraud and enables prompt prevention of damage.

[1530] The "means for detecting the start of a call" is a function for automatically detecting the moment when a user starts a call on the telephone.

[1531] "Means for streaming the voice of a call to a server in real time" refers to a function that digitizes the voice data during a call in real time and transmits it to a server via the Internet.

[1532] "Means for converting voice to text" refers to voice recognition technology that analyzes received voice data and converts it into text data.

[1533] "Means for analyzing text to determine the possibility of fraud" refers to a function that analyzes text data converted from speech and detects specific fraudulent keywords and phrases.

[1534] "Means for recording phone calls when there is a high possibility of fraud" refers to a function that records and saves as a digital audio file any phone call that is determined to be highly likely to be fraudulent as a result of analysis.

[1535] "Means for sending notifications in the event of possible fraud" is a function that sends a warning to the user and designated notification recipients when a call with a high risk of fraud is detected.

[1536] The "emotion engine that analyzes user emotions in real time" is a technology that analyzes voice characteristics (e.g., tone, speed, volume) during a call and recognizes the user's emotional state (e.g., stress, urgency) in real time.

[1537] A "generative AI model" is an artificial intelligence model that has been trained in advance with large amounts of data, and is a technology used to detect specific keywords and phrases with high accuracy.

[1538] "Means of precisely assessing the likelihood of fraud by integrating an emotion engine" is a function that integrates the text analysis results of the generative AI model with the emotion analysis results of the emotion engine to comprehensively determine the likelihood of fraud and improve its accuracy.

[1539] The "means for receiving notifications and allowing users to check the content of fraudulent calls" is an interface function that allows users to check the warning notifications they receive and view the details of calls suspected to be fraudulent.

[1540] System program generation

[1541] Hardware and Software Use

[1542] The system for implementing this invention mainly consists of a server and a user's smartphone. The server is equipped with a high-speed processor and a large-capacity SSD, and runs a speech recognition model (e.g., Google Speech-to-Text), a sentiment analysis engine (e.g., IBM Watson Tone Analyzer), and a generative AI model (e.g., OpenAI's GPT-4).

[1543] A smartphone uses the following hardware and software:

[1544] Microphone: Captures call audio.

[1545] Communications module: Streams call audio to the server in real time.

[1546] Internet connection: Allows for high speed transmission of data.

[1547] A natural language description of the program's processing

[1548] Detecting the start of a call

[1549] When a user starts a call on the phone, the smartphone application detects the call start event and notifies the server, which receives the notification and begins preparing for real-time analysis.

[1550] Streaming call audio

[1551] The user's smartphone picks up the call audio with a microphone, converts it into digital data, and sends this data over the Internet to a server, which receives the audio streaming data in real time.

[1552] Speech-to-text

[1553] The server converts the received voice data into text using a speech recognition model (e.g., Google Speech-to-Text), which is then used for subsequent analysis.

[1554] Text and Sentiment Analysis

[1555] A server-based generative AI model (e.g., GPT-4) analyzes the text data obtained from the speech recognition module to detect specific fraudulent keywords and phrases. At the same time, a sentiment analysis engine (e.g., IBM Watson Tone Analyzer) analyzes voice characteristics to recognize the user's emotions in real time. This process is carried out using advanced algorithms.

[1556] Determining the likelihood of fraud

[1557] The server combines the results of the generative AI model and the sentiment analysis engine to precisely assess the likelihood of fraud. It scores the fraud risk by combining specific keywords and the user's emotional state (e.g., high stress). If it determines there is a high likelihood of fraud, the server records the call and stores it in secure storage.

[1558] Warning notice

[1559] The server sends a warning to the user and pre-designated notification recipients (e.g., family members) for calls that are determined to be highly likely to be fraudulent. This notification can be done by push notification, email, SMS, or other means, and will send a message such as, "Signs of fraud have been detected in the content of your call."

[1560] Examples of concrete examples and prompts

[1561] When an elderly person receives a scam call pretending to be their grandchild, the emotion analysis engine detects that the tone and speed of the elderly person's voice warrants attention. The generative AI model picks up phrases such as "transfer money." This causes the server to assess the fraud risk as high, and recording begins. A notification is also sent to the family saying, "There is a possibility that your mother's call is a scam."

[1562] Prompt Sentence Examples

[1563] "Detect potential fraudulent elements in the following phone calls."

[1564] "Assess the user's stress level based on tone of voice and emotional analysis."

[1565] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1566] Step 1:

[1567] Detect the start of a call

[1568] Input: The user initiates a call.

[1569] The server receives a call start event from the smartphone application and detects the start of a call, which allows the server to prepare for real-time analysis.

[1570] Output: Call start information.

[1571] Specific behavior: When a user starts a call on their smartphone, the application catches the event and sends a call start signal to the server.

[1572] Step 2:

[1573] Streaming call audio

[1574] Input: Smartphone call audio.

[1575] The smartphone picks up the call audio through a microphone, converts it into digital data, and streams this data to a server over the internet.

[1576] Output: Real-time audio data.

[1577] How it works: The smartphone's microphone picks up sound, converts the sound signal into digital packets in real time, and sends them to a server. Because this data is sent in real time, it can be analyzed without delay.

[1578] Step 3:

[1579] Speech-to-text

[1580] Input: Real-time audio data.

[1581] The server sends the received voice data to a speech recognition model (e.g., Google Speech-to-Text) and converts the voice into text.

[1582] Output: Text data.

[1583] How it works: A server-based speech recognition model analyzes the audio data and converts it into corresponding text data, which is then used in subsequent analysis steps.

[1584] Step 4:

[1585] Text and Sentiment Analysis

[1586] Input: Text data of speech recognition results.

[1587] A generative AI model (e.g., GPT-4) analyzes text data to detect specific fraudulent keywords and phrases, while an emotion engine (e.g., IBM Watson Tone Analyzer) analyzes voice characteristics to assess user sentiment in real time.

[1588] Output: Deception probability and emotional state assessment results.

[1589] How it works: A generative AI model examines text to identify keywords and phrases related to fraud, while an emotion engine analyzes the tone, rate, and volume of speech to identify emotional states.

[1590] Step 5:

[1591] Determining the likelihood of fraud

[1592] Input: Deception likelihood and emotional state assessment results.

[1593] The server combines the results of the generative AI model and the emotion engine to assess the likelihood of fraud, and if the fraud risk is determined to be high, the call is recorded.

[1594] Output: Fraud risk score and audio recording.

[1595] How it works: The server combines the results of text analysis and sentiment analysis to score the risk of fraud. If the score exceeds a threshold, the call is recorded and stored in a secure location.

[1596] Step 6:

[1597] Warning notice

[1598] Input: Fraud risk score and audio recording.

[1599] The server will send a warning to the user and designated contacts if fraud is likely.

[1600] Output: A warning message.

[1601] What happens: The server uses the notification system to send a warning message to the user and their family members via push notification, email, SMS, etc., such as "Possible fraud detected."

[1602] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1603] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1604] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1605] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1606] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1607] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1608] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1609] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1610] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1611] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1612] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1613] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1614] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1615] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1616] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1617] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1618] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1619] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1620] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1621] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1622] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1623] The following is further disclosed regarding the above embodiment.

[1624] (Claim 1)

[1625] means for detecting the start of a call;

[1626] a means for streaming the audio of the call in real time to a server;

[1627] a means for converting speech to text;

[1628] A means of analyzing text to determine potential fraud;

[1629] A means of recording calls in cases where fraud is likely; and

[1630] a means of notifying you of potential fraud;

[1631] A system including:

[1632] (Claim 2)

[1633] 10. The system of claim 1, wherein the means for determining likelihood of fraud includes a generative AI model for detecting specific keywords or phrases.

[1634] (Claim 3)

[1635] 10. The system of claim 1, further comprising means for receiving a notification and allowing a user to verify the fraudulent call.

[1636] "Example 1"

[1637] (Claim 1)

[1638] means for detecting the start of a call;

[1639] a means for streaming the audio of the call in real time to a server;

[1640] a means for converting speech to text;

[1641] A means of analyzing text to determine potential fraud;

[1642] A means of recording calls in cases where fraud is likely; and

[1643] a means of notifying you of potential fraud;

[1644] A system including:

[1645] (Claim 2)

[1646] 10. The system of claim 1, wherein the means for determining likelihood of fraud includes a model for detecting specific keywords or phrases.

[1647] (Claim 3)

[1648] 10. The system of claim 1, further comprising means for receiving a notification and for a user to verify the fraudulent call.

[1649] "Application Example 1"

[1650] (Claim 1)

[1651] means for detecting the start of a call;

[1652] a means for streaming the audio of the call in real time to a server;

[1653] a means for converting speech to text;

[1654] A means of analyzing text to determine potential fraud;

[1655] A means of recording calls in cases where fraud is likely; and

[1656] a means of notifying you of potential fraud;

[1657] means for analyzing text of the call content, the means including a generative AI model;

[1658] A means for sending a notice of suspected fraud to a notification contact if fraud is likely;

[1659] A system including:

[1660] (Claim 2)

[1661] 10. The system of claim 1, comprising a generative AI model for detecting specific keywords and phrases.

[1662] (Claim 3)

[1663] 10. The system of claim 1, further comprising means for receiving a notification and for a user to verify the fraudulent call.

[1664] "Example 2: Combining Emotion Engines"

[1665] (Claim 1)

[1666] means for detecting the start of a call;

[1667] means for streaming the audio of the call in real time to a central processing unit;

[1668] a means for converting speech to text;

[1669] A means of analyzing text to determine the likelihood of fraud and user sentiment;

[1670] A means of recording calls in cases where fraud is likely; and

[1671] a means of notifying you of potential fraud;

[1672] A system including:

[1673] (Claim 2)

[1674] 10. The system of claim 1, wherein the means for determining likelihood of fraud includes a generative language model for detecting specific keywords and phrases and a sentiment analyzer for recognizing user sentiment.

[1675] (Claim 3)

[1676] 10. The system of claim 1, further comprising means for receiving a notification and allowing a user to verify the fraudulent call.

[1677] "Application example 2 when combining emotion engines"

[1678] (Claim 1)

[1679] means for detecting the start of a call;

[1680] a means for streaming the audio of the call in real time to a server;

[1681] a means for converting speech to text;

[1682] A means of analyzing text to determine potential fraud;

[1683] A means of recording calls in cases where fraud is likely; and

[1684] a means of notifying you of potential fraud;

[1685] It includes an emotion engine that analyzes user emotions in real time,

[1686] A means to integrate generative AI models and sentiment engines to refine the assessment of potential fraud;

[1687] A system including:

[1688] (Claim 2)

[1689] 10. The system of claim 1, wherein the means for determining likelihood of fraud includes a generative AI model for detecting specific keywords and phrases, and further includes an emotion engine for analyzing user emotion from voice characteristics.

[1690] (Claim 3)

[1691] 10. The system of claim 1, further comprising means for receiving notification and allowing a user to verify the fraudulent call, as well as means for storing the recorded call in secure storage. [Explanation of symbols]

[1692] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for detecting the start of a call; a means for streaming the audio of the call in real time to a server; a means for converting speech to text; A means of analyzing text to determine potential fraud; A means of recording calls in cases where fraud is likely; and a means of notifying you of potential fraud; A system including:

2. 10. The system of claim 1, wherein the means for determining likelihood of fraud includes a generative AI model for detecting specific keywords or phrases.

3. 10. The system of claim 1, further comprising means for receiving notification and allowing a user to verify the fraudulent call.

Images