System

Abstract

A system is provided.SOLUTION: A system comprising: a server for analyzing voice data received from a telephone terminal; a voice recognition means for converting the voice data to text; an analysis means for determining a likelihood of special fraud from the text data; a means for generating an alert message based on the determination; and a notification means for transmitting the alert message to a principal and an emergency contact.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent years, special fraud methods have become more sophisticated, and the damage caused by them is increasing. Vulnerable groups, such as the elderly, are particularly susceptible to fraud, so it is important to prevent this from happening. As fraud methods become more sophisticated, traditional warning methods and alerts alone are insufficient, and more effective countermeasures are needed. The purpose of this invention is to reduce the damage caused by special fraud by providing a method and system that can detect telephone fraud methods in real time and issue warnings quickly. [Means for solving the problem]

[0005] The present invention provides a system including a server that analyzes voice data received from a telephone terminal, a voice recognition unit that converts the voice data into text, an analysis unit that determines the possibility of special fraud from the text data, a unit that generates an alert message based on the determination, and a notification unit that sends the alert message to the caller and an emergency contact. Furthermore, the server includes a unit that automatically forwards calls from numbers not registered in the phone book, enabling prioritized analysis of calls with a high risk of fraud. Furthermore, the system includes a unit that quickly sends an alert message using a push notification service, enabling immediate warnings to relevant parties. Furthermore, the analysis unit detects keywords from the call content and determines whether fraud is suspected, achieving highly accurate fraud detection. Finally, the telephone terminal includes a unit that retains the call content only if the call is determined to be an alert, thereby protecting privacy.

[0006] "Server" means a central processing unit that analyzes voice data received from telephone terminals and generates and transmits alert messages.

[0007] "Speech recognition means" refers to a technique or device that converts received voice data into text data.

[0008] "Analysis means" refers to a technology or process that analyzes text data and determines the possibility of special fraud.

[0009] An "alert message generating means" is a technology or device that automatically generates a summary of the call content or a warning message when it is determined that fraud is suspected.

[0010] "Notification means" refers to the technology or service used to transmit the generated alert message to the individual and their emergency contacts.

[0011] A "telephone terminal" is a device that receives calls and transfers call data to a server.

[0012] "Keyword detection" is the process by which the analysis means extracts specific phrases or words from the call content.

[0013] "Push Notification Service" means an internet-based communication method for instantly sending alert messages to devices.

[0014] "Emergency contacts" are contacts other than the person himself / herself, such as family members or security companies, who should receive alert messages.

[0015] "Automatic forwarding means" is a function that automatically forwards calls from numbers not registered in the phone book to the server. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The present invention relates to a system that analyzes calls received from telephone terminals in real time and issues an alert if a special fraud is suspected. This system works in cooperation with a server, a terminal, and a user.

[0038] Server-side implementation

[0039] The server receives voice data sent from the terminal in real time. The voice data is converted into text data by a voice recognition means. For example, if someone says over the phone, "Your son has been in a traffic accident and needs urgent surgery," the content is recorded as text: "Your son has been in a traffic accident and needs urgent surgery."

[0040] Next, an analysis means analyzes this text data to determine whether it is likely to be fraud. In this case, keywords such as "traffic accident" and "surgery costs" are detected, and it is determined that there is a high possibility of fraud.

[0041] The server then uses an alert message generator to create a summary of the potentially fraudulent call. For example, it could summarize the call as "This call is about a person claiming to be your son who was in a car accident and needs money for surgery." An alert message is then sent to the person and their emergency contacts via a notification mechanism. Specifically, the message could read, "Mom's phone has received a potentially fraudulent call. Please check immediately."

[0042] Terminal side embodiment

[0043] When a device receives a call from a number not registered in the phone book, it automatically forwards the call to the server. At this time, it appears to the user as if the call were normal, but since analysis is done on the server, user interaction is minimal.

[0044] When a notification from the server arrives at the device, the device notifies the user of the content via push notification. By receiving the notification, the user can check a summary of the call content.

[0045] User's embodiment

[0046] If a user receives a notification from the server during a call, they can open the app and check a summary of the call. For example, they might see something like, "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely to be fraud, they can quickly end the call and report it to the police or an emergency contact.

[0047] Specific examples

[0048] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, the device automatically transfers the contents of the call to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If the server determines that there is a high possibility of fraud, it generates an alert message and sends it to Person A and his emergency contacts. Person A receives the notification during the call, checks the content, and determines that it is a fraud. By ending the call and reporting it to the police, the fraud can be prevented.

[0049] As described above, the present invention provides a specific method and system for detecting special telephone frauds in real time and responding quickly.

[0050] The processing flow will be explained below.

[0051] Server-side processing steps

[0052] Step 1:

[0053] The server receives a call start signal from the terminal. When the terminal detects a call from an unregistered number and begins preparations to transfer the call to the server, a call start notification is sent to the server.

[0054] Step 2:

[0055] The server receives the voice data in real time. When a call is started, the device sends the voice data to the server in streaming format. The server stores the data in a buffer.

[0056] Step 3:

[0057] The server activates the speech recognition means and converts the received speech data into text. For example, "Your son was in a traffic accident and needs urgent medical expenses."

[0058] Step 4:

[0059] The server uses analytical means to analyze the text data and detects specific fraud-related keywords (e.g., "traffic accident," "surgery costs," etc.). If such keywords are detected, it is determined that there is a suspicion of special fraud.

[0060] Step 5:

[0061] The server generates an alert message based on the text data, such as a summary of the call, such as "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery."

[0062] Step 6:

[0063] The server uses a notification method to send an alert message to the individual and their emergency contacts, such as "There was a potentially fraudulent call to your mother's phone. Please check it immediately."

[0064] Terminal processing steps

[0065] Step 1:

[0066] The device receives a call from a number that is not registered in the phone book. When a call comes in from an unregistered number, the call is automatically forwarded to the server.

[0067] Step 2:

[0068] The device sends call data to the server in real time. The call audio is recorded on the device and transferred to the server in streaming format.

[0069] Step 3:

[0070] The device waits for and receives alert notifications from the server. If the server determines that fraud is suspected and generates an alert, the notification is sent to the device.

[0071] Step 4:

[0072] The device will display an alert message to the user. When a push notification is received, the notification will say, "Warning: There has been a potentially fraudulent call. Please check the details."

[0073] User processing steps

[0074] Step 1:

[0075] The user receives a notification on the device. When an alert notification arrives on the device during a call, it is visible to the user.

[0076] Step 2:

[0077] The user opens the app and checks the summary of the call. From the summary displayed in the app, the user understands that "This call is about a person claiming to be my son who was in a traffic accident and needs money for surgery."

[0078] Step 3:

[0079] The user should end the call and, if necessary, contact the police or emergency contacts. If the user suspects fraud, they should immediately end the call and take appropriate action, such as reporting it to the police.

[0080] In this way, a system is realized in which the server, terminal, and user work together to reduce the risk of special fraud.

[0081] Example 1

[0082] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0083] In recent years, the number of victims of special telephone fraud has been increasing, with many elderly people falling victim to these frauds. To solve this problem, a system is needed that monitors the content of phone calls in real time and immediately issues an alert if it is determined that there is a high possibility of fraud. However, currently, there is no adequate system in place to achieve this kind of real-time analysis, making it difficult to take effective countermeasures. Therefore, the objective of this invention is to provide a system that automatically analyzes the content of phone calls and quickly issues an alert if fraud is suspected.

[0084] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0085] In this invention, the server includes means for analyzing voice data received from a telephone terminal in real time, means for converting the voice data into text, means for determining the possibility of special fraud from the text data, means for generating a summary of the call based on the determination, and means for sending the summary of the call to the person in question and an emergency contact, thereby making it possible to detect the possibility of fraud during a call in real time and to notify the person promptly.

[0086] A "server" is a computer system that receives the voice data, analyzes it, and processes any necessary notifications.

[0087] A "telephone terminal" is a communication device for making voice calls, and has the function of transferring voice data to a server.

[0088] "Voice data" refers to digital voice information of the contents of a call received from a telephone terminal.

[0089] "Speech recognition means" refers to technology or software for converting voice data into text data.

[0090] "Text data" is character string information generated from voice data by voice recognition means.

[0091] "Analysis means" refers to technology or software used to analyze text data and determine the possibility of fraud.

[0092] An "alert message" is a notification containing warning information about a call that is likely to be fraudulent.

[0093] "Notification Method" refers to the technology or software used to send an alert message to you and your emergency contacts.

[0094] A "call summary" is text information that briefly summarizes the contents of a call that is likely to be fraudulent.

[0095] "Push notification service" is a communication technology for sending messages from a server to a device in real time.

[0096] MODE FOR CARRYING OUT THE INVENTION

[0097] The present invention relates to a system that analyzes calls received from telephone terminals in real time and issues an alert if a special fraud is suspected. This system functions in cooperation with a server, telephone terminals, and users.

[0098] Server-side implementation

[0099] The server first receives voice data sent from the telephone terminal in real time. The voice data is converted into text data using a speech recognition method such as Google Cloud Speech-to-Text API. For example, if a caller says, "Your son was in a traffic accident and needs urgent surgery," the server converts the content into text that reads, "Your son was in a traffic accident and needs urgent surgery."

[0100] Next, the server analyzes the text data using Python's natural language processing libraries, NLTK and SpaCy. This analysis detects keywords such as "traffic accident" and "surgery costs," and determines whether there is a suspicion of fraud. In this case, it is determined that there is a high suspicion of fraud.

[0101] The server then uses an alert message generator to create a summary of the potentially fraudulent call. For example, it could summarize the call as "This call is about someone claiming to be your son who was in a car accident and needs money for surgery." The server then uses a notification mechanism to send an alert message to the person and their emergency contact. Specifically, the message could read, "Mom's phone has received a potentially fraudulent call. Please check immediately."

[0102] Terminal side embodiment

[0103] When a phone device receives a call from a number not registered in the phone book, it automatically forwards the call to the server. Although it appears to the user as if the call is normal, analysis is performed on the server, so the user's operation is minimal. In addition, when a notification from the server arrives on the device, the device notifies the user of the content via a push notification. Upon receiving the notification, the user can check a summary of the call content.

[0104] User's embodiment

[0105] If a user receives a notification from the server during a call, they can open the app and check a summary of the call. For example, they might see something like, "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely to be fraud, they can quickly end the call and report it to the police or an emergency contact.

[0106] Specific examples

[0107] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, the telephone terminal automatically transfers the contents of the call to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If the server determines that there is a high possibility of fraud, it generates an alert message and sends it to Person A and his emergency contacts. Person A receives the notification during the call, checks the content, and determines that it is a fraud. By ending the call and reporting it to the police, the fraud can be prevented.

[0108] Prompt Sentence Examples

[0109] "Please tell me how this system can be used to detect special fraudulent phone calls in real time. For example, please explain in detail how you would respond if someone said, 'My son was in a traffic accident.'"

[0110] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0111] Step 1:

[0112] The device receives a call from a number that is not registered in the phone book. The input is the incoming phone number, and the output is that the call from that phone number is automatically recorded and forwarded to the server. Specifically, the device records the contents of the call in real time and sends the recorded data to the server.

[0113] Step 2:

[0114] The server receives the voice data transferred from the terminal in real time. The input is the voice data sent from the terminal, and the output is the voice data stored on the server. Specifically, the server stores the voice data in an appropriate format.

[0115] Step 3:

[0116] The server converts the voice data into text data using the Google Cloud Speech-to-Text API. The input is the stored voice data, and the output is the generated text data. Specifically, the server sends the voice data to the speech recognition API and saves the received text data.

[0117] Step 4:

[0118] The server analyzes the text data to determine whether it is likely to be fraud. The input is the generated text data, and the output is the analysis result. Specifically, the server uses Python's NLTK and SpaCy to extract keywords such as "traffic accident" and "surgery costs" from the text data, and determines the possibility of fraud based on these keywords.

[0119] Step 5:

[0120] The server generates a summary of the call if there is a high possibility of fraud. The inputs are the analysis results and text data, and the output is a summary of the call. Specifically, the server uses an alert message generation means to create a summary such as, "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery."

[0121] Step 6:

[0122] The server sends the generated alert message to the person and their emergency contact. The input is a summary of the call, and the output is the alert message. Specifically, the server uses the notification method to send a message via a push notification service saying, "There was a call to your mother's phone that is likely fraudulent. Please check it immediately."

[0123] Step 7:

[0124] The device receives an alert message sent from the server and notifies the user via a push notification. The input is the alert message from the server, and the output is a notification displayed on the user's device screen. Specifically, the device displays the received notification as a pop-up on the screen.

[0125] Step 8:

[0126] The user sees the alert message displayed on the device. The input is a summary of the call displayed on the device, and the output provides information that the user can use to determine the possibility of fraud. The user opens the app and sees the detailed summary of the call.

[0127] Step 9:

[0128] If the user determines that there is a high possibility of fraud, they will end the call and report it to the police or emergency contacts. The input is the user's fraud judgment, and the output is a report. Specifically, the user will end the call promptly and contact the police or emergency contacts.

[0129] (Application example 1)

[0130] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0131] Conventional telephone fraud prevention systems analyze voice data to determine the possibility of fraud, but have the problem that users cannot receive visual warnings of fraud while on the phone. Also, if the smart device owner is physically away from the phone, it is difficult to respond quickly. This has led to the problem that fraud victims cannot be sufficiently prevented from falling victim to fraud.

[0132] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0133] In this invention, the server includes means for analyzing voice data received from a telephone terminal, means for converting the voice data into text, means for determining the possibility of special fraud from the text data, means for generating an alert message based on the determination, means for sending the alert message to the user and an emergency contact, and means for displaying a visual alert on a smart device. This allows the user to instantly receive a visual fraud warning during a call, enabling a prompt response even when the user is physically away from the telephone.

[0134] A "telephone terminal" is a communication device for making voice calls.

[0135] "Audio data" refers to data that records or transmits audio signals in digital form.

[0136] A "server" is a computer system for processing and managing data.

[0137] "Speech recognition means" is a technology that converts voice data into text data.

[0138] "Text data" is data expressed as a string of characters.

[0139] "Analysis means" refers to the technology that analyzes given data and extracts specific information or patterns.

[0140] An "alert message" is a message that contains warning or notification content.

[0141] A "notification means" is a technique for conveying information to a particular device or user.

[0142] A "smart device" is a highly functional electronic device equipped with communication and computer functions.

[0143] A "visual alert" is a method of displaying warnings or notifications using visual elements.

[0144] MODE FOR CARRYING OUT THE INVENTION

[0145] The present invention provides a system that analyzes calls received from a telephone terminal in real time and issues an alert if a special fraud is suspected. The following describes in detail the embodiments of the present invention.

[0146] Server-side implementation

[0147] The server receives voice data from the telephone terminal in real time. The voice data is converted into text data by a voice recognition means. For example, if a caller says, "Your son was in a traffic accident and urgently needs surgery," the call is recorded as text: "Your son was in a traffic accident and urgently needs surgery." The analysis means then analyzes the text data to determine whether it is likely fraudulent. In this case, keywords such as "traffic accident" and "surgery costs" are detected, indicating a high likelihood of fraud. The server then uses an alert message generation means to create a summary of the potentially fraudulent call. For example, the summary might read, "This call is about someone claiming to be your son being in a traffic accident and needing surgery." The notification means then sends an alert message to the caller and their emergency contact. Specifically, the message might read, "There has been a potentially fraudulent call to your mother's phone. Please check immediately." A visual alert display on the smart device is also provided. This allows for a prompt response even when the caller is away from the phone.

[0148] Terminal side embodiment

[0149] When a device receives a call from a number not registered in the phone book, it automatically forwards the call to the server. At this time, it appears to the user as if the call were normal, but since analysis is performed on the server, user operation is minimal. In addition, when a notification from the server arrives on the device or smart device, the device notifies the user of the content via push notification. Upon receiving the notification, the user can check a summary of the call content.

[0150] User's embodiment

[0151] When a user receives a notification from the server during a call, they open the application and check a summary of the call. For example, they might see something like, "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely fraudulent, they can quickly end the call and report it to the police or emergency contacts. A visual alert displayed on the smart device also allows for quick action.

[0152] Hardware and Software

[0153] Hardware:

[0154] server

[0155] telephone terminal

[0156] Smart devices (e.g., smart glasses, smartphones)

[0157] software:

[0158] Speech recognition: speech_recognition

[0159] Translation: googletrans

[0160] Text analysis: textblob

[0161] Specific examples

[0162] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for surgery costs. When this call begins, the device automatically transfers the call contents to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If the server determines that the call is likely to be fraudulent, it generates an alert message and sends it to Person A and his / her emergency contacts. Person A receives the notification during the call, checks the content, and determines that it is fraudulent. By ending the call and reporting it to the police, the fraud victim can be prevented. In addition, a visual alert is displayed on smart devices such as smart glasses, so the warning can be received immediately even if the person is away from the phone.

[0163] Prompt Sentence Examples

[0164] If a user wants to add fraud keywords to a generative AI model, here is an example prompt:

[0165] "Add new scam keyword: {new keyword}"

[0166] for example:

[0167] "Add new fraud keywords: bank account, PIN, payment due date"

[0168] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0169] Step 1:

[0170] The terminal detects that there is an incoming call to the telephone terminal. If the call is from a number not registered in the phone book, the terminal transfers the call to the server, which then sends the voice data as input data to be provided to the server.

[0171] Step 2:

[0172] The server converts the received voice data into text data in real time using a speech recognition tool. This conversion process uses speech recognition software (speech_recognition). The input is voice data, and the output is text data that expresses the voice in characters.

[0173] Step 3:

[0174] The server analyzes the converted text data using analytical means to determine the possibility of special fraud. This analysis uses a text analysis library (textblob) and a translation library (googletrans). The input is text data, and data calculations are performed to detect fraud keywords and evaluate their likelihood. The output is an evaluation result indicating the possibility of fraud.

[0175] Step 4:

[0176] If the server determines that fraud is likely, it generates an alert message using an alert message generation means. Specifically, it generates a text message summarizing the details of the fraud. The input is the evaluation result and the text data that may be fraudulent, and the output is the alert message.

[0177] Step 5:

[0178] The server sends the generated alert message to the individual and their emergency contacts using a notification mechanism. Additionally, a visual alert is displayed on the smart device (e.g., smart glasses, smartphone). The notification mechanism uses a push notification service. The input is the alert message, and the output is the notification transmission result.

[0179] Step 6:

[0180] The user receives a push notification from their terminal or smart device during a call and checks the content. The user opens the application and reads a summary of the call to determine whether it is fraudulent. The input is the alert message, and the output is the user's decision.

[0181] Step 7:

[0182] If the user determines that there is a high possibility of fraud, they can quickly end the call and report it to the police or an emergency contact. This makes it possible to prevent actual fraud from occurring. The input is the user's judgment, and the output is the reporting action.

[0183] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0184] The present invention relates to a system that analyzes calls received from telephone terminals in real time and issues an alert if a special fraud is suspected. In particular, the present invention improves the accuracy of fraud detection by combining voice data analysis with an emotion engine that recognizes the user's emotions.

[0185] Server-side implementation

[0186] The server receives voice data sent from the terminal in real time. The voice data is converted into text data by a voice recognition means. For example, if someone says over the phone, "Your son has been in a traffic accident and needs urgent surgery," the content is recorded as text: "Your son has been in a traffic accident and needs urgent surgery."

[0187] Next, an analysis means analyzes this text data to determine whether it is likely to be fraud. In this case, keywords such as "traffic accident" and "surgery costs" are detected, and it is determined that there is a high possibility of fraud.

[0188] Additionally, the server is equipped with an emotion engine that analyzes the user's emotional state from the voice data. The emotion engine detects emotions such as stress or anxiety, and provides this to the analysis means as auxiliary information to further strengthen suspicion of fraud.

[0189] The server then uses an alert message generator to create a summary of the potentially fraudulent call. For example, it could summarize the call as "This call is about a person claiming to be your son who was in a car accident and needs money for surgery." An alert message is then sent to the person and their emergency contacts via a notification mechanism. Specifically, the message could read, "Mom's phone has received a potentially fraudulent call. Please check immediately."

[0190] Terminal side embodiment

[0191] When a device receives a call from a number not registered in the phone book, it automatically forwards the call to the server. At this time, it appears to the user as if the call were normal, but since analysis is done on the server, user interaction is minimal.

[0192] When a notification from the server arrives at the device, the device notifies the user of the content via push notification. By receiving the notification, the user can check a summary of the call content.

[0193] User's embodiment

[0194] If a user receives a notification from the server during a call, they can open the app and check a summary of the call. For example, they might see something like, "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely to be fraud, they can quickly end the call and report it to the police or an emergency contact.

[0195] Specific examples

[0196] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, the device automatically transfers the contents of the call to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If the server determines that there is a high possibility of fraud, it uses an emotion engine to analyze the user's emotional state, and if suspicious emotions are confirmed, it incorporates this as supplementary information. Finally, an alert message is generated and sent to Person A and their emergency contact. After Person A receives the notification during the call and checks the content, he or she determines that it is a fraud, ends the call, and reports it to the police, thereby preventing fraud from occurring.

[0197] As described above, the present invention provides a specific method and system for detecting special telephone frauds in real time and responding quickly by making full use of an emotion engine.

[0198] The processing flow will be explained below.

[0199] Server-side processing steps

[0200] Step 1:

[0201] The server receives a call start signal from the terminal. When the terminal detects a call from an unregistered number and begins preparations to transfer the call to the server, a call start notification is sent to the server.

[0202] Step 2:

[0203] The server receives the voice data in real time. When a call is started, the device sends the voice data to the server in streaming format. The server stores the data in a buffer.

[0204] Step 3:

[0205] The server activates the speech recognition means and converts the received voice data into text. For example, the contents of the call may be converted into text such as "Your son has been in a traffic accident and urgently needs money for surgery."

[0206] Step 4:

[0207] The server uses an analytical method to analyze the text data, detect fraud-related keywords (e.g., "traffic accident," "surgery costs," etc.), and determine whether there is any suspicion of fraud.

[0208] Step 5:

[0209] The server activates an emotion engine to analyze the user's emotional state (e.g., anxiety, stress, etc.) from the received voice data. This emotional state is treated as auxiliary information to strengthen suspicion of fraud.

[0210] Step 6:

[0211] The server generates an alert message based on the text data and the results of sentiment analysis. For example, a summary such as "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery" is generated.

[0212] Step 7:

[0213] The server will use a notification method to send an alert message to the individual and their emergency contacts, specifically a message such as, "There was a likely fraudulent call to your mother's phone. Please check immediately."

[0214] Terminal processing steps

[0215] Step 1:

[0216] The device receives a call from a number that is not registered in the phone book. When a call comes in from an unregistered number, the call is automatically forwarded to the server. When the call starts, the voice data is sent to the server.

[0217] Step 2:

[0218] The device sends call data to the server in real time. The call audio is recorded on the device and transferred to the server in streaming format.

[0219] Step 3:

[0220] The device waits for and receives alert notifications from the server. If the server determines that fraud is suspected and generates an alert, the notification is sent to the device.

[0221] Step 4:

[0222] The device will display an alert message to the user. When a push notification is received, the notification will say, "Warning: There has been a potentially fraudulent call. Please check the details."

[0223] User processing steps

[0224] Step 1:

[0225] The user receives a notification on the device. When an alert notification arrives on the device during a call, it is visible to the user.

[0226] Step 2:

[0227] The user opens the app and checks the summary of the call. The summary displayed in the app reads, "This call is from someone claiming to be your son and he needs money for surgery after a traffic accident."

[0228] Step 3:

[0229] The user should end the call and, if necessary, contact the police or emergency contacts. If there is a strong suspicion of fraud, the user should immediately end the call and take appropriate action, such as reporting the call to the police.

[0230] In this way, a system is realized in which the server, terminal, and user work together to reduce the risk of special fraud. By combining this system with an emotion engine, the accuracy of fraud detection can be improved, enabling more effective fraud countermeasures.

[0231] Example 2

[0232] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0233] Special fraud methods are becoming more sophisticated every year, and it is extremely difficult for victims to recognize fraud in real time, especially in cases of fraud over the phone. As a result, many people fall victim to fraud and suffer huge losses. In addition, it is necessary to accurately analyze the content of fraudulent calls and detect the emotional state of the victim, but conventional systems are inadequate in dealing with this important aspect.

[0234] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0235] In this invention, the server includes means for analyzing voice data received from a telephone terminal, means for converting the voice data into text, means for determining the possibility of special fraud from the text data, means for analyzing the emotional state using the analysis means, means for generating an alert message based on the determination, and means for sending the alert message to the caller and their emergency contacts. This makes it possible to analyze the content of fraudulent calls in real time and quickly issue an alert if there is a high possibility of fraud. Furthermore, by analyzing the victim's emotional state, the accuracy of fraud detection can be improved, helping to prevent damage before it occurs.

[0236] "Server" refers to a computer system that analyzes and manages voice data received from telephone terminals, and determines and notifies users of the possibility of fraud.

[0237] "Voice data" refers to the voice signal received from the telephone terminal, which is the subject of analysis.

[0238] "Speech recognition means" refers to the process or technology that converts voice data into text data, and a speech recognition engine is an example of this.

[0239] "Text data" refers to data in sentence format converted by a speech recognition means, and serves as the basis for analysis.

[0240] "Analysis methods" refers to the processes and technologies that analyze text data, detect specific keywords and phrases, and determine the possibility of special fraud.

[0241] "Emotion analysis means" refers to processes or technologies that analyze a user's emotional state from voice or text data and provide supplemental information that reinforces the possibility of fraud.

[0242] "Alert Message" refers to a warning message summarizing the content of a call that is determined to be highly likely to be fraudulent.

[0243] "Message generation means" refers to the process or technology for creating an alert message based on the results of the analysis means and the sentiment analysis means.

[0244] "Means of Notification" refers to the process or technology by which the generated alert message is sent to the individual and their emergency contacts.

[0245] "Push notification service" refers to a service that sends messages from a server to a user's device in real time.

[0246] "Telephone terminal" refers to a device for making voice calls.

[0247] This invention relates to a system that analyzes calls received from telephone terminals in real time and issues an alert if a special fraud is suspected. Specifically, it improves the accuracy of fraud detection by combining voice data analysis with an emotion analysis engine that recognizes user emotions.

[0248] Server-side implementation

[0249] The server receives voice data sent from the telephone terminal in real time. This voice data is converted into text data using a speech recognition means. For example, the voice data is converted into text data using the Google Cloud Speech-to-Text API. As a specific example, if someone says over the phone, "Your son has been in a traffic accident and needs urgent medical expenses," the content of the call is recorded as text as "Your son has been in a traffic accident and needs urgent medical expenses."

[0250] The server then uses a natural language processing engine, such as IBM Watson Natural Language Understanding, to analyze the text data. This process determines whether there is a possibility of fraud. For example, if certain keywords, such as "traffic accident" or "surgery costs," are detected, it is determined that there is a high possibility of fraud.

[0251] The server is also equipped with an emotion analysis engine that analyzes the user's emotional state from the voice data. For example, Emotion AI is used to detect emotions such as stress and anxiety. This emotional data is provided to the analysis means as auxiliary information to further increase the likelihood of fraud.

[0252] Furthermore, the server generates an alert message based on the result of the judgment. The alert message generation means creates a summary such as, "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery." The notification means then sends the alert message to the person and their emergency contact. Specifically, the message sent may read, "There has been a call to your mother's phone that is likely to be fraudulent. Please check immediately."

[0253] Terminal side embodiment

[0254] When a device receives a call from a number not registered in the phone book, it automatically forwards the call to the server. At this time, it appears to the user as if the call were normal, but since analysis is done on the server, user interaction is minimal.

[0255] In addition, when a notification from the server arrives on the device, the device will notify the user of the content via push notification. When the notification is received, the user can easily check the content of the notification. For example, a prompt message may be sent saying, "There was a call to your mother's phone that is likely to be fraudulent. Please check it immediately."

[0256] User's embodiment

[0257] When a user receives a notification from the server during a call, they can open a dedicated app and check a summary of the call. For example, a summary such as "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery" may be displayed. If the user determines based on this summary that the call is likely to be fraud, they can immediately end the call and report it to the police or an emergency contact.

[0258] Specific examples

[0259] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, Person A's device automatically transfers the voice call to the server. The server receives the voice data and converts it into text using the Google Cloud Speech-to-Text API. It then analyzes the text data using IBM Watson Natural Language Understanding to detect keywords such as "traffic accident" and "surgery costs." If the server determines that there is a high possibility of fraud, it uses Emotion AI to analyze the user's emotional state, and if emotions such as stress or anxiety are detected, it incorporates this as supplemental information.

[0260] Finally, the server generates an alert message and sends it to Person A and his / her emergency contacts. After Person A receives the notification during the call and checks the summary, he / she determines that the call is likely a scam, immediately ends the call, and reports it to the police. This prevents the scam from occurring.

[0261] As described above, the present invention provides a specific method and system for detecting special telephone frauds in real time and responding quickly by making full use of an emotion analysis engine.

[0262] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0263] Step 1:

[0264] When a terminal receives a call from a number not registered in the phone book, it automatically transfers the call contents to the server. It receives voice data from the telephone terminal as input and sends this voice data to the server as output. From the user's side, it appears as if a normal call is being made, but in reality, the call contents are being sent to the server in real time.

[0265] Step 2:

[0266] The server receives voice data sent from the terminal in real time. It receives voice data from the terminal as input and passes it to the voice recognition means as output. Specifically, the voice data is sent to the server in streaming format, and the server supplies it to the voice recognition engine.

[0267] Step 3:

[0268] The server converts the received voice data into text data using a speech recognition method (for example, Google Cloud Speech-to-Text API). As input, it receives real-time voice data and converts it into text data. The output is the converted text data. For example, a message such as "Your son has been in a traffic accident and urgently needs money for surgery" may be recorded as text.

[0269] Step 4:

[0270] The server analyzes the text data using analytical means (for example, IBM Watson Natural Language Understanding). It receives the text data as input and detects specific keywords or phrases contained within it. The output is a determination result as to whether or not the keyword or phrase is included. For example, if keywords such as "traffic accident" or "surgery costs" are detected, it determines that the call is likely to be fraudulent.

[0271] Step 5:

[0272] The server uses emotion analysis means (e.g., Emotion AI) to analyze the user's emotional state from the voice data. As input, it receives the original voice data and text data and detects emotions such as stress and anxiety. The output is the detected emotion data. For example, if high anxiety or stress is detected, this is used as auxiliary information to strengthen the possibility of fraud.

[0273] Step 6:

[0274] The server generates an alert message based on the analysis results and sentiment analysis results. It receives the text analysis results and sentiment analysis data as input, and creates a summarized warning message using the alert message generation means. For example, it generates a summary such as "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery." The output is this summary message.

[0275] Step 7:

[0276] The server sends the generated alert message to the user and emergency contacts using a notification method. It receives the generated alert message as input and sends it as output to the user and emergency contacts via email, SMS, or push notification services. Specifically, it sends a message saying, "There was a likely fraudulent call on your mother's phone. Please check it immediately."

[0277] Step 8:

[0278] If a user receives an alert notification during a call, the user opens a dedicated app and checks a summary of the call. The input is receiving the alert notification and displaying the notification content within the application. The output is summary information that is displayed on the user's screen. For example, it might say, "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery."

[0279] Step 9:

[0280] If the user determines based on this summary information that the call is likely to be fraudulent, they can immediately end the call and report it to the police or emergency contacts. The input is to check the alert notification and summary information and take appropriate action. The output is to end the call and report it to the police. For example, a fraudulent call can be reported using the "Report" button provided in the dedicated app.

[0281] (Application example 2)

[0282] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0283] The methods of special frauds using telephone terminals are becoming more sophisticated, and existing systems may have difficulty detecting them. Furthermore, systems that ignore the user's emotional state have the problem of being unable to adequately determine the possibility of fraud. Therefore, there is a need for a system that can detect fraud with greater accuracy by analyzing both voice data and the user's emotional state.

[0284] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0285] In this invention, the server includes means for analyzing voice data received from a telephone terminal, means for converting the voice data into text, means for determining the possibility of special fraud from the text data, means for generating an alert message, means for sending the alert message to the user and an emergency contact, emotion analysis means for detecting the user's emotional state from the voice data, and means for using the output of the emotion analysis means as supplementary information for the determination. This makes it possible to analyze both the voice data and the user's emotional state and detect the possibility of fraud with higher accuracy.

[0286] A "server" is a central computer system that analyzes voice data received from telephone terminals and determines the possibility of fraud.

[0287] "Speech recognition means" refers to a technique or device for converting received voice data into text data.

[0288] "Analysis means" refers to a technology or device for analyzing text data to determine the possibility of special fraud.

[0289] The "alert message generating means" is a technology or device that, when the analysis means determines that there is a possibility of fraud, generates a message to notify the user or emergency contact based on that information.

[0290] "Notification means" refers to the technology or device used to send the generated alert message to the individual and their emergency contacts.

[0291] An "emotion analysis means" is a technique or device for detecting a user's emotional state (for example, stress or anxiety) from voice data.

[0292] "Supplementary information" is information for incorporating data on the user's emotional state obtained by the emotion analysis means into the determination process of the analysis means.

[0293] The present invention provides a detailed description of a system that detects special frauds in real time via telephone terminals and sends alerts to users and emergency contacts.

[0294] Server embodiment

[0295] The server receives voice data sent from the telephone terminal in real time. The hardware required is a server machine with a high-performance processor and sufficient memory, and the software uses the "Google Speech Recognition API" as the voice recognition engine, the "EmotionalEngine" as the emotion analysis engine, and the "AlertSystem" as the alert system. The server's processing procedure is as follows:

[0296] First, the server receives the voice data and converts it into text data using a voice recognition device. For example, if the caller says, "My son was in a traffic accident and needs urgent surgery money," the content of the call will be recorded as text data.

[0297] Next, an analysis means analyzes this text data to determine the possibility of fraud. At this stage, specific keywords such as "traffic accident" or "surgery costs" are detected. The server also has an emotion analysis means that detects the user's emotional state (e.g., stress or anxiety) from the voice data. This information is provided to the analysis means as supplementary information to further strengthen suspicion of fraud.

[0298] Based on the analysis results, the alert message generating means creates a summary of the potentially fraudulent call, and the notification means sends an alert message to the user and emergency contacts. Specifically, a message such as "There has been a potentially fraudulent call to your mother's phone. Please check it immediately" is sent.

[0299] Terminal embodiment

[0300] The device has a function that automatically forwards calls from numbers not registered in the phone book to the server. At this time, it appears to the user as if they are making a normal call, but the content of the call is analyzed by the server, so the user's operation is kept to a minimum.

[0301] When a notification from the server arrives at the device, the device notifies the user of the content via push notification. By receiving the notification, the user can check a summary of the call content.

[0302] User's embodiment

[0303] If a user receives a notification from the server during a call, they can open the "Fraud Guard" application installed on their smartphone and check a summary of the call. For example, it might say, "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely to be fraud, they can quickly end the call and report it to the police or an emergency contact.

[0304] Specific examples

[0305] As a concrete example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, the device automatically transfers the contents of the call to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If it determines that there is a high possibility of fraud, the server performs an emotion analysis, and if it confirms that the user is in a state of high stress, it incorporates this as supplementary information. Finally, an alert message is generated and sent to Person A and their emergency contacts.

[0306] Prompt Sentence Examples

[0307] Example prompts to input to a generative AI model:

[0308] Perform voice analysis, and if the converted text contains words like "traffic accident" or "surgery costs," use emotion analysis to check for stress or anxiety, and generate a program that sends an alert to the user if there is a suspicion of fraud.

[0309] As described above, the present invention provides a specific method and system for detecting special telephone frauds in real time and responding quickly by making full use of an emotion engine.

[0310] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0311] Step 1:

[0312] The telephone terminal receives the incoming call.

[0313] Input: Audio data when a phone call is received.

[0314] Specific operation: When a call is received from an unregistered number, the device starts the call and transfers the voice data to the server in real time.

[0315] Output: Audio data transferred to the server.

[0316] Step 2:

[0317] The server receives the audio data.

[0318] Input: Audio data transferred from the device.

[0319] Specific operation: The server receives audio data in real time using a high-performance data stream processing device.

[0320] Output: The audio data is saved to the server's storage device.

[0321] Step 3:

[0322] The server converts the voice data into text data.

[0323] Input: Stored audio data.

[0324] What it does: Uses the Google Speech Recognition API to convert audio data into corresponding text data.

[0325] Output: Text data.

[0326] Step 4:

[0327] The server analyzes the text data and determines whether it is a special fraud.

[0328] Input: The converted text data.

[0329] Specific operation: Searches for specific keywords (e.g., "traffic accident" or "surgery costs") in the text data. If the conditions are met, it is determined to be a possible fraud.

[0330] Output: Determination result regarding likelihood of fraud.

[0331] Step 5:

[0332] The server detects the user's emotional state from the voice data.

[0333] Input: Received audio data.

[0334] Specific behavior: Uses EmotionalEngine to detect the user's emotional state (e.g., stress or anxiety) from audio data.

[0335] Output: Detected user emotional state data.

[0336] Step 6:

[0337] The server uses the emotional state data as supplementary information for analysis.

[0338] Input: Emotional state data and analysis results.

[0339] What it does: If the user is in a state of high stress or anxiety, that information is used to supplement the analysis results and make a stronger judgment about the likelihood of fraud.

[0340] Output: The final fraud verdict taking into account the supplementary information.

[0341] Step 7:

[0342] The server generates an alert message.

[0343] Input: Final fraud determination result.

[0344] Specific action: Based on the judgment result, a specific alert message (e.g., "This call may be fraudulent. Please check the content.") is automatically generated.

[0345] Output: The alert message.

[0346] Step 8:

[0347] The server sends an alert message via a notification means.

[0348] Input: The generated alert message.

[0349] Specific operation: Uses push notification service to send an alert message to the user and their emergency contacts.

[0350] Output: Sending an alert message to the user and emergency contacts.

[0351] Step 9:

[0352] The user acknowledges the alert message.

[0353] Input: The received alert message.

[0354] Specific actions: Open the Fraud Guard app installed on your smartphone and check the call summary and alert message.

[0355] Output: The user becomes aware of the possibility of fraud and takes necessary measures.

[0356] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0357] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0358] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0359] [Second embodiment]

[0360] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0361] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0362] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0363] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0364] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0365] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0366] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0367] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0368] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0369] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0370] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0371] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0372] The present invention relates to a system that analyzes calls received from telephone terminals in real time and issues an alert if a special fraud is suspected. This system works in cooperation with a server, a terminal, and a user.

[0373] Server-side implementation

[0374] The server receives voice data sent from the terminal in real time. The voice data is converted into text data by a voice recognition means. For example, if someone says over the phone, "Your son has been in a traffic accident and needs urgent surgery," the content is recorded as text: "Your son has been in a traffic accident and needs urgent surgery."

[0375] Next, an analysis means analyzes this text data to determine whether it is likely to be fraud. In this case, keywords such as "traffic accident" and "surgery costs" are detected, and it is determined that there is a high possibility of fraud.

[0376] The server then uses an alert message generator to create a summary of the potentially fraudulent call. For example, it could summarize the call as "This call is about a person claiming to be your son who was in a car accident and needs money for surgery." An alert message is then sent to the person and their emergency contacts via a notification mechanism. Specifically, the message could read, "Mom's phone has received a potentially fraudulent call. Please check immediately."

[0377] Terminal side embodiment

[0378] When a call comes in from a number not registered in the phone book, the device automatically forwards the call to the server. At this time, it appears to the user as if the call is normal, but since analysis is done on the server, user operation is minimal.

[0379] When a notification from the server arrives at the device, the device notifies the user of the content via push notification. By receiving the notification, the user can check a summary of the call content.

[0380] User's embodiment

[0381] If a user receives a notification from the server during a call, they can open the app and check a summary of the call. For example, they might see something like, "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely to be fraud, they can quickly end the call and report it to the police or an emergency contact.

[0382] Specific examples

[0383] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, the device automatically transfers the contents of the call to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If the server determines that there is a high possibility of fraud, it generates an alert message and sends it to Person A and his emergency contacts. Person A receives the notification during the call, checks the content, and determines that it is a fraud. By ending the call and reporting it to the police, the fraud can be prevented.

[0384] As described above, the present invention provides a specific method and system for detecting special telephone frauds in real time and responding quickly.

[0385] The processing flow will be explained below.

[0386] Server-side processing steps

[0387] Step 1:

[0388] The server receives a call start signal from the terminal. When the terminal detects a call from an unregistered number and begins preparations to transfer the call to the server, a call start notification is sent to the server.

[0389] Step 2:

[0390] The server receives the voice data in real time. When a call is started, the device sends the voice data in streaming format to the server, which stores it in a buffer.

[0391] Step 3:

[0392] The server activates the speech recognition means and converts the received speech data into text. For example, "Your son was in a traffic accident and needs urgent medical expenses."

[0393] Step 4:

[0394] The server uses analytical means to analyze the text data and detects specific fraud-related keywords (e.g., "traffic accident," "surgery costs," etc.). If such keywords are detected, it is determined that there is a suspicion of special fraud.

[0395] Step 5:

[0396] The server generates an alert message based on the text data, such as a summary of the call, such as "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery."

[0397] Step 6:

[0398] The server uses a notification method to send an alert message to the individual and their emergency contacts, such as "There was a potentially fraudulent call to your mother's phone. Please check it immediately."

[0399] Terminal processing steps

[0400] Step 1:

[0401] The device receives a call from a number that is not registered in the phone book. When a call comes in from an unregistered number, the call is automatically forwarded to the server.

[0402] Step 2:

[0403] The device sends call data to the server in real time. The call audio is recorded on the device and transferred to the server in streaming format.

[0404] Step 3:

[0405] The device waits for and receives alert notifications from the server. If the server determines that fraud is suspected and generates an alert, the notification is sent to the device.

[0406] Step 4:

[0407] The device will display an alert message to the user. When a push notification is received, the notification will say, "Warning: There has been a potentially fraudulent call. Please check the details."

[0408] User processing steps

[0409] Step 1:

[0410] The user receives a notification on the device. When an alert notification arrives on the device during a call, it is visible to the user.

[0411] Step 2:

[0412] The user opens the app and checks the summary of the call. From the summary displayed in the app, the user understands that "This call is about a person claiming to be my son who was in a traffic accident and needs money for surgery."

[0413] Step 3:

[0414] The user should end the call and, if necessary, contact the police or emergency contacts. If the user suspects fraud, they should immediately end the call and take appropriate action, such as reporting it to the police.

[0415] In this way, a system is realized in which the server, terminal, and user work together to reduce the risk of special fraud.

[0416] Example 1

[0417] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0418] In recent years, the number of victims of special telephone fraud has been increasing, with many elderly people falling victim to these frauds. To solve this problem, a system is needed that monitors the content of phone calls in real time and immediately issues an alert if it is determined that there is a high possibility of fraud. However, currently, there is no adequate system in place to achieve this kind of real-time analysis, making it difficult to take effective countermeasures. Therefore, the objective of this invention is to provide a system that automatically analyzes the content of phone calls and quickly issues an alert if fraud is suspected.

[0419] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0420] In this invention, the server includes means for analyzing voice data received from a telephone terminal in real time, means for converting the voice data into text, means for determining the possibility of special fraud from the text data, means for generating a summary of the call based on the determination, and means for sending the summary of the call to the person in question and an emergency contact, thereby making it possible to detect the possibility of fraud during a call in real time and to notify the person promptly.

[0421] A "server" is a computer system that receives the voice data, analyzes it, and processes any necessary notifications.

[0422] A "telephone terminal" is a communication device for making voice calls, and has the function of transferring voice data to a server.

[0423] "Voice data" refers to digital voice information of the contents of a call received from a telephone terminal.

[0424] "Speech recognition means" refers to technology or software for converting voice data into text data.

[0425] "Text data" is character string information generated from voice data by voice recognition means.

[0426] "Analysis means" refers to technology or software used to analyze text data and determine the possibility of fraud.

[0427] An "alert message" is a notification containing warning information about a call that is likely to be fraudulent.

[0428] "Notification Method" refers to the technology or software used to send an alert message to you and your emergency contacts.

[0429] A "call summary" is text information that briefly summarizes the contents of a call that is likely to be fraudulent.

[0430] "Push notification service" is a communication technology for sending messages from a server to a device in real time.

[0431] MODE FOR CARRYING OUT THE INVENTION

[0432] The present invention relates to a system that analyzes calls received from telephone terminals in real time and issues an alert if a special fraud is suspected. This system functions in cooperation with a server, telephone terminals, and users.

[0433] Server-side implementation

[0434] The server first receives voice data sent from the telephone terminal in real time. The voice data is converted into text data using a speech recognition method such as Google Cloud Speech-to-Text API. For example, if a caller says, "Your son was in a traffic accident and needs urgent surgery," the server converts the content into text that reads, "Your son was in a traffic accident and needs urgent surgery."

[0435] Next, the server analyzes the text data using Python's natural language processing libraries, NLTK and SpaCy. This analysis detects keywords such as "traffic accident" and "surgery costs," and determines whether there is a suspicion of fraud. In this case, it is determined that there is a high suspicion of fraud.

[0436] The server then uses an alert message generator to create a summary of the potentially fraudulent call. For example, it could summarize the call as "This call is about someone claiming to be your son who was in a car accident and needs money for surgery." The server then uses a notification mechanism to send an alert message to the person and their emergency contact. Specifically, the message could read, "Mom's phone has received a potentially fraudulent call. Please check immediately."

[0437] Terminal side embodiment

[0438] When a phone device receives a call from a number not registered in the phone book, it automatically forwards the call to the server. Although it appears to the user as if the call is normal, analysis is performed on the server, so the user's operation is minimal. In addition, when a notification from the server arrives on the device, the device notifies the user of the content via a push notification. Upon receiving the notification, the user can check a summary of the call content.

[0439] User's embodiment

[0440] If a user receives a notification from the server during a call, they can open the app and check a summary of the call. For example, they might see something like, "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely to be fraud, they can quickly end the call and report it to the police or an emergency contact.

[0441] Specific examples

[0442] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, the telephone terminal automatically transfers the contents of the call to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If the server determines that there is a high possibility of fraud, it generates an alert message and sends it to Person A and his emergency contacts. Person A receives the notification during the call, checks the content, and determines that it is a fraud. By ending the call and reporting it to the police, the fraud can be prevented.

[0443] Prompt Sentence Examples

[0444] "Please tell me how this system can be used to detect special fraudulent phone calls in real time. For example, please explain in detail how you would respond if someone said, 'My son was in a traffic accident.'"

[0445] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0446] Step 1:

[0447] The device receives a call from a number that is not registered in the phone book. The input is the incoming phone number, and the output is that the call from that phone number is automatically recorded and forwarded to the server. Specifically, the device records the contents of the call in real time and sends the recorded data to the server.

[0448] Step 2:

[0449] The server receives the voice data transferred from the terminal in real time. The input is the voice data sent from the terminal, and the output is the voice data stored on the server. Specifically, the server stores the voice data in an appropriate format.

[0450] Step 3:

[0451] The server converts the voice data into text data using the Google Cloud Speech-to-Text API. The input is the stored voice data, and the output is the generated text data. Specifically, the server sends the voice data to the speech recognition API and saves the received text data.

[0452] Step 4:

[0453] The server analyzes the text data to determine whether it is likely to be fraud. The input is the generated text data, and the output is the analysis result. Specifically, the server uses Python's NLTK and SpaCy to extract keywords such as "traffic accident" and "surgery costs" from the text data, and determines the possibility of fraud based on these keywords.

[0454] Step 5:

[0455] The server generates a summary of the call if there is a high possibility of fraud. The inputs are the analysis results and text data, and the output is a summary of the call. Specifically, the server uses an alert message generation means to create a summary such as, "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery."

[0456] Step 6:

[0457] The server sends the generated alert message to the person and their emergency contact. The input is a summary of the call, and the output is the alert message. Specifically, the server uses the notification method to send a message via a push notification service saying, "There was a call to your mother's phone that is likely fraudulent. Please check it immediately."

[0458] Step 7:

[0459] The device receives an alert message sent from the server and notifies the user via a push notification. The input is the alert message from the server, and the output is a notification displayed on the user's device screen. Specifically, the device displays the received notification as a pop-up on the screen.

[0460] Step 8:

[0461] The user sees the alert message displayed on the device. The input is a summary of the call displayed on the device, and the output provides information that the user can use to determine the possibility of fraud. The user opens the app and sees the detailed summary of the call.

[0462] Step 9:

[0463] If the user determines that there is a high possibility of fraud, they will end the call and report it to the police or emergency contacts. The input is the user's fraud judgment, and the output is a report. Specifically, the user will end the call promptly and contact the police or emergency contacts.

[0464] (Application example 1)

[0465] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0466] Conventional telephone fraud prevention systems analyze voice data to determine the possibility of fraud, but have the problem that users cannot receive visual warnings of fraud while on the phone. Also, if the smart device owner is physically away from the phone, it is difficult to respond quickly. This has led to the problem that fraud victims cannot be sufficiently prevented from falling victim to fraud.

[0467] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0468] In this invention, the server includes means for analyzing voice data received from a telephone terminal, means for converting the voice data into text, means for determining the possibility of special fraud from the text data, means for generating an alert message based on the determination, means for sending the alert message to the user and an emergency contact, and means for displaying a visual alert on a smart device. This allows the user to instantly receive a visual fraud warning during a call, enabling a prompt response even when the user is physically away from the telephone.

[0469] A "telephone terminal" is a communication device for making voice calls.

[0470] "Audio data" refers to data that records or transmits audio signals in digital form.

[0471] A "server" is a computer system for processing and managing data.

[0472] "Speech recognition means" is a technology that converts voice data into text data.

[0473] "Text data" is data expressed as a string of characters.

[0474] "Analysis means" refers to the technology that analyzes given data and extracts specific information or patterns.

[0475] An "alert message" is a message that contains warning or notification content.

[0476] A "notification means" is a technique for conveying information to a particular device or user.

[0477] A "smart device" is a highly functional electronic device equipped with communication and computer functions.

[0478] A "visual alert" is a method of displaying warnings or notifications using visual elements.

[0479] MODE FOR CARRYING OUT THE INVENTION

[0480] The present invention provides a system that analyzes calls received from a telephone terminal in real time and issues an alert if a special fraud is suspected. The following describes in detail the embodiments of the present invention.

[0481] Server-side implementation

[0482] The server receives voice data from the telephone terminal in real time. The voice data is converted into text data by a voice recognition means. For example, if a caller says, "Your son was in a traffic accident and urgently needs surgery," the call is recorded as text: "Your son was in a traffic accident and urgently needs surgery." The analysis means then analyzes the text data to determine whether it is likely fraudulent. In this case, keywords such as "traffic accident" and "surgery costs" are detected, indicating a high likelihood of fraud. The server then uses an alert message generation means to create a summary of the potentially fraudulent call. For example, the summary might read, "This call is about someone claiming to be your son being in a traffic accident and needing surgery." The notification means then sends an alert message to the caller and their emergency contact. Specifically, the message might read, "There has been a potentially fraudulent call to your mother's phone. Please check immediately." A visual alert display on the smart device is also provided. This allows for a prompt response even when the caller is away from the phone.

[0483] Terminal side embodiment

[0484] When a device receives a call from a number not registered in the phone book, it automatically forwards the call to the server. At this time, it appears to the user as if the call were normal, but since analysis is performed on the server, user operation is minimal. In addition, when a notification from the server arrives on the device or smart device, the device notifies the user of the content via push notification. Upon receiving the notification, the user can check a summary of the call content.

[0485] User's embodiment

[0486] When a user receives a notification from the server during a call, they open the application and check a summary of the call. For example, they might see something like, "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely fraudulent, they can quickly end the call and report it to the police or emergency contacts. A visual alert displayed on the smart device also allows for quick action.

[0487] Hardware and Software

[0488] Hardware:

[0489] server

[0490] telephone terminal

[0491] Smart devices (e.g., smart glasses, smartphones)

[0492] software:

[0493] Speech recognition: speech_recognition

[0494] Translation: googletrans

[0495] Text analysis: textblob

[0496] Specific examples

[0497] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for surgery costs. When this call begins, the device automatically transfers the call contents to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If the server determines that the call is likely to be fraudulent, it generates an alert message and sends it to Person A and his / her emergency contacts. Person A receives the notification during the call, checks the content, and determines that it is fraudulent. By ending the call and reporting it to the police, the fraud victim can be prevented. In addition, a visual alert is displayed on smart devices such as smart glasses, so the warning can be received immediately even if the person is away from the phone.

[0498] Prompt Sentence Examples

[0499] If a user wants to add fraud keywords to a generative AI model, here is an example prompt:

[0500] "Add new scam keyword: {new keyword}"

[0501] for example:

[0502] "Add new fraud keywords: bank account, PIN, payment due date"

[0503] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0504] Step 1:

[0505] The terminal detects that there is an incoming call to the telephone terminal. If the call is from a number not registered in the phone book, the terminal transfers the call to the server, which then sends the voice data as input data to be provided to the server.

[0506] Step 2:

[0507] The server converts the received voice data into text data in real time using a speech recognition tool. This conversion process uses speech recognition software (speech_recognition). The input is voice data, and the output is text data that expresses the voice in characters.

[0508] Step 3:

[0509] The server analyzes the converted text data using analytical means to determine the possibility of special fraud. This analysis uses a text analysis library (textblob) and a translation library (googletrans). The input is text data, and data calculations are performed to detect fraud keywords and evaluate their likelihood. The output is an evaluation result indicating the possibility of fraud.

[0510] Step 4:

[0511] If the server determines that fraud is likely, it generates an alert message using an alert message generation means. Specifically, it generates a text message summarizing the details of the fraud. The input is the evaluation result and the text data that may be fraudulent, and the output is the alert message.

[0512] Step 5:

[0513] The server sends the generated alert message to the individual and their emergency contacts using a notification mechanism. Additionally, a visual alert is displayed on the smart device (e.g., smart glasses, smartphone). The notification mechanism uses a push notification service. The input is the alert message, and the output is the notification transmission result.

[0514] Step 6:

[0515] The user receives a push notification from their terminal or smart device during a call and checks the content. The user opens the application and reads a summary of the call to determine whether it is fraudulent. The input is the alert message, and the output is the user's decision.

[0516] Step 7:

[0517] If the user determines that there is a high possibility of fraud, they can quickly end the call and report it to the police or an emergency contact. This makes it possible to prevent actual fraud from occurring. The input is the user's judgment, and the output is the reporting action.

[0518] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0519] The present invention relates to a system that analyzes calls received from telephone terminals in real time and issues an alert if a special fraud is suspected. In particular, the present invention improves the accuracy of fraud detection by combining voice data analysis with an emotion engine that recognizes the user's emotions.

[0520] Server-side implementation

[0521] The server receives voice data sent from the terminal in real time. The voice data is converted into text data by a voice recognition means. For example, if someone says over the phone, "Your son has been in a traffic accident and needs urgent surgery," the content is recorded as text: "Your son has been in a traffic accident and needs urgent surgery."

[0522] Next, an analysis means analyzes this text data to determine whether it is likely to be fraud. In this case, keywords such as "traffic accident" and "surgery costs" are detected, and it is determined that there is a high possibility of fraud.

[0523] Additionally, the server is equipped with an emotion engine that analyzes the user's emotional state from the voice data. The emotion engine detects emotions such as stress or anxiety, and provides this to the analysis means as auxiliary information to further strengthen suspicion of fraud.

[0524] The server then uses an alert message generator to create a summary of the potentially fraudulent call. For example, it could summarize the call as "This call is about a person claiming to be your son who was in a car accident and needs money for surgery." An alert message is then sent to the person and their emergency contacts via a notification mechanism. Specifically, the message could read, "Mom's phone has received a potentially fraudulent call. Please check immediately."

[0525] Terminal side embodiment

[0526] When a device receives a call from a number not registered in the phone book, it automatically forwards the call to the server. At this time, it appears to the user as if the call were normal, but since analysis is done on the server, user interaction is minimal.

[0527] When a notification from the server arrives at the device, the device notifies the user of the content via push notification. By receiving the notification, the user can check a summary of the call content.

[0528] User's embodiment

[0529] If a user receives a notification from the server during a call, they can open the app and check a summary of the call. For example, they might see something like, "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely to be fraud, they can quickly end the call and report it to the police or an emergency contact.

[0530] Specific examples

[0531] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, the device automatically transfers the contents of the call to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If the server determines that there is a high possibility of fraud, it uses an emotion engine to analyze the user's emotional state, and if suspicious emotions are confirmed, it incorporates this as supplementary information. Finally, an alert message is generated and sent to Person A and their emergency contact. After Person A receives the notification during the call and checks the content, he or she determines that it is a fraud, ends the call, and reports it to the police, thereby preventing fraud from occurring.

[0532] As described above, the present invention provides a specific method and system for detecting special telephone frauds in real time and responding quickly by making full use of an emotion engine.

[0533] The processing flow will be explained below.

[0534] Server-side processing steps

[0535] Step 1:

[0536] The server receives a call start signal from the terminal. When the terminal detects a call from an unregistered number and begins preparations to transfer the call to the server, a call start notification is sent to the server.

[0537] Step 2:

[0538] The server receives the voice data in real time. When a call is started, the device sends the voice data to the server in streaming format. The server stores the data in a buffer.

[0539] Step 3:

[0540] The server activates the speech recognition means and converts the received voice data into text. For example, the contents of the call may be converted into text such as "Your son has been in a traffic accident and urgently needs money for surgery."

[0541] Step 4:

[0542] The server uses an analytical method to analyze the text data, detect fraud-related keywords (e.g., "traffic accident," "surgery costs," etc.), and determine whether there is any suspicion of fraud.

[0543] Step 5:

[0544] The server activates an emotion engine to analyze the user's emotional state (e.g., anxiety, stress, etc.) from the received voice data. This emotional state is treated as auxiliary information to strengthen suspicion of fraud.

[0545] Step 6:

[0546] The server generates an alert message based on the text data and the results of sentiment analysis. For example, a summary such as "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery" is generated.

[0547] Step 7:

[0548] The server will use a notification method to send an alert message to the individual and their emergency contacts, specifically a message such as, "There was a likely fraudulent call to your mother's phone. Please check immediately."

[0549] Terminal processing steps

[0550] Step 1:

[0551] The device receives a call from a number that is not registered in the phone book. When a call comes in from an unregistered number, the call is automatically forwarded to the server. When the call starts, the voice data is sent to the server.

[0552] Step 2:

[0553] The device sends call data to the server in real time. The call audio is recorded on the device and transferred to the server in streaming format.

[0554] Step 3:

[0555] The device waits for and receives alert notifications from the server. If the server determines that fraud is suspected and generates an alert, the notification is sent to the device.

[0556] Step 4:

[0557] The device will display an alert message to the user. When a push notification is received, the notification will say, "Warning: There has been a potentially fraudulent call. Please check the details."

[0558] User processing steps

[0559] Step 1:

[0560] The user receives a notification on the device. When an alert notification arrives on the device during a call, it is visible to the user.

[0561] Step 2:

[0562] The user opens the app and checks the summary of the call. The summary displayed in the app reads, "This call is from someone claiming to be your son and he needs money for surgery after a traffic accident."

[0563] Step 3:

[0564] The user should end the call and, if necessary, contact the police or emergency contacts. If there is a strong suspicion of fraud, the user should immediately end the call and take appropriate action, such as reporting the call to the police.

[0565] In this way, a system is realized in which the server, terminal, and user work together to reduce the risk of special fraud. By combining this system with an emotion engine, the accuracy of fraud detection can be improved, enabling more effective fraud countermeasures.

[0566] Example 2

[0567] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0568] Special fraud methods are becoming more sophisticated every year, and it is extremely difficult for victims to recognize fraud in real time, especially in cases of fraud over the phone. As a result, many people fall victim to fraud and suffer huge losses. In addition, it is necessary to accurately analyze the content of fraudulent calls and detect the emotional state of the victim, but conventional systems are inadequate in dealing with this important aspect.

[0569] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0570] In this invention, the server includes means for analyzing voice data received from a telephone terminal, means for converting the voice data into text, means for determining the possibility of special fraud from the text data, means for analyzing the emotional state using the analysis means, means for generating an alert message based on the determination, and means for sending the alert message to the caller and their emergency contacts. This makes it possible to analyze the content of fraudulent calls in real time and quickly issue an alert if there is a high possibility of fraud. Furthermore, by analyzing the victim's emotional state, the accuracy of fraud detection can be improved, helping to prevent damage before it occurs.

[0571] "Server" refers to a computer system that analyzes and manages voice data received from telephone terminals, and determines and notifies users of the possibility of fraud.

[0572] "Voice data" refers to the voice signal received from the telephone terminal, which is the subject of analysis.

[0573] "Speech recognition means" refers to the process or technology that converts voice data into text data, and a speech recognition engine is an example of this.

[0574] "Text data" refers to data in sentence format converted by a speech recognition means, and serves as the basis for analysis.

[0575] "Analysis methods" refers to the processes and technologies that analyze text data, detect specific keywords and phrases, and determine the possibility of special fraud.

[0576] "Emotion analysis means" refers to processes or technologies that analyze a user's emotional state from voice or text data and provide supplemental information that reinforces the possibility of fraud.

[0577] "Alert Message" refers to a warning message summarizing the content of a call that is determined to be highly likely to be fraudulent.

[0578] "Message generation means" refers to the process or technology for creating an alert message based on the results of the analysis means and the sentiment analysis means.

[0579] "Means of Notification" refers to the process or technology by which the generated alert message is sent to the individual and their emergency contacts.

[0580] "Push notification service" refers to a service that sends messages from a server to a user's device in real time.

[0581] "Telephone terminal" refers to a device for making voice calls.

[0582] This invention relates to a system that analyzes calls received from telephone terminals in real time and issues an alert if a special fraud is suspected. Specifically, it improves the accuracy of fraud detection by combining voice data analysis with an emotion analysis engine that recognizes user emotions.

[0583] Server-side implementation

[0584] The server receives voice data sent from the telephone terminal in real time. This voice data is converted into text data using a speech recognition means. For example, the voice data is converted into text data using the Google Cloud Speech-to-Text API. As a specific example, if someone says over the phone, "Your son has been in a traffic accident and needs urgent medical expenses," the content of the call is recorded as text as "Your son has been in a traffic accident and needs urgent medical expenses."

[0585] The server then uses a natural language processing engine, such as IBM Watson Natural Language Understanding, to analyze the text data. This process determines whether there is a possibility of fraud. For example, if certain keywords, such as "traffic accident" or "surgery costs," are detected, it is determined that there is a high possibility of fraud.

[0586] The server is also equipped with an emotion analysis engine that analyzes the user's emotional state from the voice data. For example, Emotion AI is used to detect emotions such as stress and anxiety. This emotional data is provided to the analysis means as auxiliary information to further increase the likelihood of fraud.

[0587] Furthermore, the server generates an alert message based on the result of the judgment. The alert message generation means creates a summary such as, "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery." The notification means then sends the alert message to the person and their emergency contact. Specifically, the message sent may read, "There has been a call to your mother's phone that is likely to be fraudulent. Please check immediately."

[0588] Terminal side embodiment

[0589] When a device receives a call from a number not registered in the phone book, it automatically forwards the call to the server. At this time, it appears to the user as if the call were normal, but since analysis is done on the server, user interaction is minimal.

[0590] In addition, when a notification from the server arrives on the device, the device will notify the user of the content via push notification. When the notification is received, the user can easily check the content of the notification. For example, a prompt message may be sent saying, "There was a call to your mother's phone that is likely to be fraudulent. Please check it immediately."

[0591] User's embodiment

[0592] When a user receives a notification from the server during a call, they can open a dedicated app and check a summary of the call. For example, a summary such as "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery" may be displayed. If the user determines based on this summary that the call is likely to be fraud, they can immediately end the call and report it to the police or an emergency contact.

[0593] Specific examples

[0594] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, Person A's device automatically transfers the voice call to the server. The server receives the voice data and converts it into text using the Google Cloud Speech-to-Text API. It then analyzes the text data using IBM Watson Natural Language Understanding to detect keywords such as "traffic accident" and "surgery costs." If the server determines that there is a high possibility of fraud, it uses Emotion AI to analyze the user's emotional state, and if emotions such as stress or anxiety are detected, it incorporates this as supplemental information.

[0595] Finally, the server generates an alert message and sends it to Person A and his / her emergency contacts. After Person A receives the notification during the call and checks the summary, he / she determines that the call is likely a scam, immediately ends the call, and reports it to the police. This prevents the scam from occurring.

[0596] As described above, the present invention provides a specific method and system for detecting special telephone frauds in real time and responding quickly by making full use of an emotion analysis engine.

[0597] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0598] Step 1:

[0599] When a terminal receives a call from a number not registered in the phone book, it automatically transfers the call contents to the server. It receives voice data from the telephone terminal as input and sends this voice data to the server as output. From the user's side, it appears as if a normal call is being made, but in reality, the call contents are being sent to the server in real time.

[0600] Step 2:

[0601] The server receives voice data sent from the terminal in real time. It receives voice data from the terminal as input and passes it to the voice recognition means as output. Specifically, the voice data is sent to the server in streaming format, and the server supplies it to the voice recognition engine.

[0602] Step 3:

[0603] The server converts the received voice data into text data using a speech recognition method (for example, Google Cloud Speech-to-Text API). As input, it receives real-time voice data and converts it into text data. The output is the converted text data. For example, a message such as "Your son has been in a traffic accident and urgently needs money for surgery" may be recorded as text.

[0604] Step 4:

[0605] The server analyzes the text data using analytical means (for example, IBM Watson Natural Language Understanding). It receives the text data as input and detects specific keywords or phrases contained within it. The output is a determination result as to whether or not the keyword or phrase is included. For example, if keywords such as "traffic accident" or "surgery costs" are detected, it determines that the call is likely to be fraudulent.

[0606] Step 5:

[0607] The server uses emotion analysis means (e.g., Emotion AI) to analyze the user's emotional state from the voice data. As input, it receives the original voice data and text data and detects emotions such as stress and anxiety. The output is the detected emotion data. For example, if high anxiety or stress is detected, this is used as auxiliary information to strengthen the possibility of fraud.

[0608] Step 6:

[0609] The server generates an alert message based on the analysis results and sentiment analysis results. It receives the text analysis results and sentiment analysis data as input, and creates a summarized warning message using the alert message generation means. For example, it generates a summary such as "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery." The output is this summary message.

[0610] Step 7:

[0611] The server sends the generated alert message to the user and emergency contacts using a notification method. It receives the generated alert message as input and sends it as output to the user and emergency contacts via email, SMS, or push notification services. Specifically, it sends a message saying, "There was a likely fraudulent call on your mother's phone. Please check it immediately."

[0612] Step 8:

[0613] If a user receives an alert notification during a call, the user opens a dedicated app and checks a summary of the call. The input is receiving the alert notification and displaying the notification content within the application. The output is summary information that is displayed on the user's screen. For example, it might say, "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery."

[0614] Step 9:

[0615] If the user determines based on this summary information that the call is likely to be fraudulent, they can immediately end the call and report it to the police or emergency contacts. The input is to check the alert notification and summary information and take appropriate action. The output is to end the call and report it to the police. For example, a fraudulent call can be reported using the "Report" button provided in the dedicated app.

[0616] (Application example 2)

[0617] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0618] The methods of special frauds using telephone terminals are becoming more sophisticated, and existing systems may have difficulty detecting them. Furthermore, systems that ignore the user's emotional state have the problem of being unable to adequately determine the possibility of fraud. Therefore, there is a need for a system that can detect fraud with greater accuracy by analyzing both voice data and the user's emotional state.

[0619] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0620] In this invention, the server includes means for analyzing voice data received from a telephone terminal, means for converting the voice data into text, means for determining the possibility of special fraud from the text data, means for generating an alert message, means for sending the alert message to the user and an emergency contact, emotion analysis means for detecting the user's emotional state from the voice data, and means for using the output of the emotion analysis means as supplementary information for the determination. This makes it possible to analyze both the voice data and the user's emotional state and detect the possibility of fraud with higher accuracy.

[0621] A "server" is a central computer system that analyzes voice data received from telephone terminals and determines the possibility of fraud.

[0622] "Speech recognition means" refers to a technique or device for converting received voice data into text data.

[0623] "Analysis means" refers to a technology or device for analyzing text data to determine the possibility of special fraud.

[0624] The "alert message generating means" is a technology or device that, when the analysis means determines that there is a possibility of fraud, generates a message to notify the user or emergency contact based on that information.

[0625] "Notification means" refers to the technology or device used to send the generated alert message to the individual and their emergency contacts.

[0626] An "emotion analysis means" is a technique or device for detecting a user's emotional state (for example, stress or anxiety) from voice data.

[0627] "Supplementary information" is information for incorporating data on the user's emotional state obtained by the emotion analysis means into the determination process of the analysis means.

[0628] The present invention provides a detailed description of a system that detects special frauds in real time via telephone terminals and sends alerts to users and emergency contacts.

[0629] Server embodiment

[0630] The server receives voice data sent from the telephone terminal in real time. The hardware required is a server machine with a high-performance processor and sufficient memory, and the software uses the "Google Speech Recognition API" as the voice recognition engine, the "EmotionalEngine" as the emotion analysis engine, and the "AlertSystem" as the alert system. The server's processing procedure is as follows:

[0631] First, the server receives the voice data and converts it into text data using a voice recognition device. For example, if the caller says, "My son was in a traffic accident and needs urgent surgery money," the content of the call will be recorded as text data.

[0632] Next, an analysis means analyzes this text data to determine the possibility of fraud. At this stage, specific keywords such as "traffic accident" or "surgery costs" are detected. The server also has an emotion analysis means that detects the user's emotional state (e.g., stress or anxiety) from the voice data. This information is provided to the analysis means as supplementary information to further strengthen suspicion of fraud.

[0633] Based on the analysis results, the alert message generating means creates a summary of the potentially fraudulent call, and the notification means sends an alert message to the user and emergency contacts. Specifically, a message such as "There has been a potentially fraudulent call to your mother's phone. Please check it immediately" is sent.

[0634] Terminal embodiment

[0635] The device has a function that automatically forwards calls from numbers not registered in the phone book to the server. At this time, it appears to the user as if they are making a normal call, but the content of the call is analyzed by the server, so the user's operation is kept to a minimum.

[0636] When a notification from the server arrives at the device, the device notifies the user of the content via push notification. By receiving the notification, the user can check a summary of the call content.

[0637] User's embodiment

[0638] If a user receives a notification from the server during a call, they can open the "Fraud Guard" application installed on their smartphone and check a summary of the call. For example, it might say, "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely to be fraud, they can quickly end the call and report it to the police or an emergency contact.

[0639] Specific examples

[0640] As a concrete example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, the device automatically transfers the contents of the call to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If it determines that there is a high possibility of fraud, the server performs an emotion analysis, and if it confirms that the user is in a state of high stress, it incorporates this as supplementary information. Finally, an alert message is generated and sent to Person A and their emergency contacts.

[0641] Prompt Sentence Examples

[0642] Example prompts to input to a generative AI model:

[0643] Perform voice analysis, and if the converted text contains words like "traffic accident" or "surgery costs," use emotion analysis to check for stress or anxiety, and generate a program that sends an alert to the user if there is a suspicion of fraud.

[0644] As described above, the present invention provides a specific method and system for detecting special telephone frauds in real time and responding quickly by making full use of an emotion engine.

[0645] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0646] Step 1:

[0647] The telephone terminal receives the incoming call.

[0648] Input: Audio data when a phone call is received.

[0649] Specific operation: When a call is received from an unregistered number, the device starts the call and transfers the voice data to the server in real time.

[0650] Output: Audio data transferred to the server.

[0651] Step 2:

[0652] The server receives the audio data.

[0653] Input: Audio data transferred from the device.

[0654] Specific operation: The server receives audio data in real time using a high-performance data stream processing device.

[0655] Output: The audio data is saved to the server's storage device.

[0656] Step 3:

[0657] The server converts the voice data into text data.

[0658] Input: Stored audio data.

[0659] What it does: Uses the Google Speech Recognition API to convert audio data into corresponding text data.

[0660] Output: Text data.

[0661] Step 4:

[0662] The server analyzes the text data and determines whether it is a special fraud.

[0663] Input: The converted text data.

[0664] Specific operation: Searches for specific keywords (e.g., "traffic accident" or "surgery costs") in the text data. If the conditions are met, it is determined to be a possible fraud.

[0665] Output: Determination result regarding likelihood of fraud.

[0666] Step 5:

[0667] The server detects the user's emotional state from the voice data.

[0668] Input: Received audio data.

[0669] Specific behavior: Uses EmotionalEngine to detect the user's emotional state (e.g., stress or anxiety) from audio data.

[0670] Output: Detected user emotional state data.

[0671] Step 6:

[0672] The server uses the emotional state data as supplementary information for analysis.

[0673] Input: Emotional state data and analysis results.

[0674] What it does: If the user is in a state of high stress or anxiety, that information is used to supplement the analysis results and make a stronger judgment about the likelihood of fraud.

[0675] Output: The final fraud verdict taking into account the supplementary information.

[0676] Step 7:

[0677] The server generates an alert message.

[0678] Input: Final fraud determination result.

[0679] Specific action: Based on the judgment result, a specific alert message (e.g., "This call may be fraudulent. Please check the content.") is automatically generated.

[0680] Output: The alert message.

[0681] Step 8:

[0682] The server sends an alert message via a notification means.

[0683] Input: The generated alert message.

[0684] Specific operation: Uses push notification service to send an alert message to the user and their emergency contacts.

[0685] Output: Sending an alert message to the user and emergency contacts.

[0686] Step 9:

[0687] The user acknowledges the alert message.

[0688] Input: The received alert message.

[0689] Specific actions: Open the Fraud Guard app installed on your smartphone and check the call summary and alert message.

[0690] Output: The user becomes aware of the possibility of fraud and takes necessary measures.

[0691] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0692] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0693] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0694] [Third embodiment]

[0695] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0696] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0697] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0698] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0699] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0700] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0701] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0702] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0703] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0704] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0705] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0706] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0707] The present invention relates to a system that analyzes calls received from telephone terminals in real time and issues an alert if a special fraud is suspected. This system works in cooperation with a server, a terminal, and a user.

[0708] Server-side implementation

[0709] The server receives voice data sent from the terminal in real time. The voice data is converted into text data by a voice recognition means. For example, if someone says over the phone, "Your son has been in a traffic accident and needs urgent surgery," the content is recorded as text: "Your son has been in a traffic accident and needs urgent surgery."

[0710] Next, an analysis means analyzes this text data to determine whether it is likely to be fraud. In this case, keywords such as "traffic accident" and "surgery costs" are detected, and it is determined that there is a high possibility of fraud.

[0711] The server then uses an alert message generator to create a summary of the potentially fraudulent call. For example, it could summarize the call as "This call is about a person claiming to be your son who was in a car accident and needs money for surgery." An alert message is then sent to the person and their emergency contacts via a notification mechanism. Specifically, the message could read, "Mom's phone has received a potentially fraudulent call. Please check immediately."

[0712] Terminal side embodiment

[0713] When a call comes in from a number not registered in the phone book, the device automatically forwards the call to the server. At this time, it appears to the user as if the call is normal, but since analysis is done on the server, user operation is minimal.

[0714] When a notification from the server arrives at the device, the device notifies the user of the content via push notification. By receiving the notification, the user can check a summary of the call content.

[0715] User's embodiment

[0716] If a user receives a notification from the server during a call, they can open the app and check a summary of the call. For example, they might see something like, "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely to be fraud, they can quickly end the call and report it to the police or an emergency contact.

[0717] Specific examples

[0718] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, the device automatically transfers the contents of the call to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If the server determines that there is a high possibility of fraud, it generates an alert message and sends it to Person A and his emergency contacts. Person A receives the notification during the call, checks the content, and determines that it is a fraud. By ending the call and reporting it to the police, the fraud can be prevented.

[0719] As described above, the present invention provides a specific method and system for detecting special telephone frauds in real time and responding quickly.

[0720] The processing flow will be explained below.

[0721] Server-side processing steps

[0722] Step 1:

[0723] The server receives a call start signal from the terminal. When the terminal detects a call from an unregistered number and begins preparations to transfer the call to the server, a call start notification is sent to the server.

[0724] Step 2:

[0725] The server receives the voice data in real time. When a call is started, the device sends the voice data in streaming format to the server, which stores it in a buffer.

[0726] Step 3:

[0727] The server activates the speech recognition means and converts the received speech data into text. For example, "Your son was in a traffic accident and needs urgent medical expenses."

[0728] Step 4:

[0729] The server uses analytical means to analyze the text data and detects specific fraud-related keywords (e.g., "traffic accident," "surgery costs," etc.). If such keywords are detected, it is determined that there is a suspicion of special fraud.

[0730] Step 5:

[0731] The server generates an alert message based on the text data, such as a summary of the call, such as "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery."

[0732] Step 6:

[0733] The server uses a notification method to send an alert message to the individual and their emergency contacts, such as "There was a potentially fraudulent call to your mother's phone. Please check it immediately."

[0734] Terminal processing steps

[0735] Step 1:

[0736] The device receives a call from a number that is not registered in the phone book. When a call comes in from an unregistered number, the call is automatically forwarded to the server.

[0737] Step 2:

[0738] The device sends call data to the server in real time. The call audio is recorded on the device and transferred to the server in streaming format.

[0739] Step 3:

[0740] The device waits for and receives alert notifications from the server. If the server determines that fraud is suspected and generates an alert, the notification is sent to the device.

[0741] Step 4:

[0742] The device will display an alert message to the user. When a push notification is received, the notification will say, "Warning: There has been a potentially fraudulent call. Please check the details."

[0743] User processing steps

[0744] Step 1:

[0745] The user receives a notification on the device. When an alert notification arrives on the device during a call, it is visible to the user.

[0746] Step 2:

[0747] The user opens the app and checks the summary of the call. From the summary displayed in the app, the user understands that "This call is about a person claiming to be my son who was in a traffic accident and needs money for surgery."

[0748] Step 3:

[0749] The user should end the call and, if necessary, contact the police or emergency contacts. If the user suspects fraud, they should immediately end the call and take appropriate action, such as reporting it to the police.

[0750] In this way, a system is realized in which the server, terminal, and user work together to reduce the risk of special fraud.

[0751] Example 1

[0752] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0753] In recent years, the number of victims of special telephone fraud has been increasing, with many elderly people falling victim to these frauds. To solve this problem, a system is needed that monitors the content of phone calls in real time and immediately issues an alert if it is determined that there is a high possibility of fraud. However, currently, there is no adequate system in place to achieve this kind of real-time analysis, making it difficult to take effective countermeasures. Therefore, the objective of this invention is to provide a system that automatically analyzes the content of phone calls and quickly issues an alert if fraud is suspected.

[0754] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0755] In this invention, the server includes means for analyzing voice data received from a telephone terminal in real time, means for converting the voice data into text, means for determining the possibility of special fraud from the text data, means for generating a summary of the call based on the determination, and means for sending the summary of the call to the person in question and an emergency contact, thereby making it possible to detect the possibility of fraud during a call in real time and to notify the person promptly.

[0756] A "server" is a computer system that receives the voice data, analyzes it, and processes any necessary notifications.

[0757] A "telephone terminal" is a communication device for making voice calls, and has the function of transferring voice data to a server.

[0758] "Voice data" refers to digital voice information of the contents of a call received from a telephone terminal.

[0759] "Speech recognition means" refers to technology or software for converting voice data into text data.

[0760] "Text data" is character string information generated from voice data by voice recognition means.

[0761] "Analysis means" refers to technology or software used to analyze text data and determine the possibility of fraud.

[0762] An "alert message" is a notification containing warning information about a call that is likely to be fraudulent.

[0763] "Notification Method" refers to the technology or software used to send an alert message to you and your emergency contacts.

[0764] A "call summary" is text information that briefly summarizes the contents of a call that is likely to be fraudulent.

[0765] "Push notification service" is a communication technology for sending messages from a server to a device in real time.

[0766] MODE FOR CARRYING OUT THE INVENTION

[0767] The present invention relates to a system that analyzes calls received from telephone terminals in real time and issues an alert if a special fraud is suspected. This system functions in cooperation with a server, telephone terminals, and users.

[0768] Server-side implementation

[0769] The server first receives voice data sent from the telephone terminal in real time. The voice data is converted into text data using a speech recognition method such as Google Cloud Speech-to-Text API. For example, if a caller says, "Your son was in a traffic accident and needs urgent surgery," the server converts the content into text that reads, "Your son was in a traffic accident and needs urgent surgery."

[0770] Next, the server analyzes the text data using Python's natural language processing libraries, NLTK and SpaCy. This analysis detects keywords such as "traffic accident" and "surgery costs," and determines whether there is a suspicion of fraud. In this case, it is determined that there is a high suspicion of fraud.

[0771] The server then uses an alert message generator to create a summary of the potentially fraudulent call. For example, it could summarize the call as "This call is about someone claiming to be your son who was in a car accident and needs money for surgery." The server then uses a notification mechanism to send an alert message to the person and their emergency contact. Specifically, the message could read, "Mom's phone has received a potentially fraudulent call. Please check immediately."

[0772] Terminal side embodiment

[0773] When a phone device receives a call from a number not registered in the phone book, it automatically forwards the call to the server. Although it appears to the user as if the call is normal, analysis is performed on the server, so the user's operation is minimal. In addition, when a notification from the server arrives on the device, the device notifies the user of the content via a push notification. Upon receiving the notification, the user can check a summary of the call content.

[0774] User's embodiment

[0775] If a user receives a notification from the server during a call, they can open the app and check a summary of the call. For example, they might see something like, "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely to be fraud, they can quickly end the call and report it to the police or an emergency contact.

[0776] Specific examples

[0777] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, the telephone terminal automatically transfers the contents of the call to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If the server determines that there is a high possibility of fraud, it generates an alert message and sends it to Person A and his emergency contacts. Person A receives the notification during the call, checks the content, and determines that it is a fraud. By ending the call and reporting it to the police, the fraud can be prevented.

[0778] Prompt Sentence Examples

[0779] "Please tell me how this system can be used to detect special telephone fraud in real time. For example, please explain in detail how you would respond if someone said, 'My son was in a traffic accident.'"

[0780] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0781] Step 1:

[0782] The device receives a call from a number that is not registered in the phone book. The input is the incoming phone number, and the output is that the call from that phone number is automatically recorded and forwarded to the server. Specifically, the device records the contents of the call in real time and sends the recorded data to the server.

[0783] Step 2:

[0784] The server receives the voice data transferred from the terminal in real time. The input is the voice data sent from the terminal, and the output is the voice data stored on the server. Specifically, the server stores the voice data in an appropriate format.

[0785] Step 3:

[0786] The server converts the voice data into text data using the Google Cloud Speech-to-Text API. The input is the stored voice data, and the output is the generated text data. Specifically, the server sends the voice data to the speech recognition API and saves the received text data.

[0787] Step 4:

[0788] The server analyzes the text data to determine whether it is likely to be fraud. The input is the generated text data, and the output is the analysis result. Specifically, the server uses Python's NLTK and SpaCy to extract keywords such as "traffic accident" and "surgery costs" from the text data, and determines the possibility of fraud based on these keywords.

[0789] Step 5:

[0790] The server generates a summary of the call if there is a high possibility of fraud. The inputs are the analysis results and text data, and the output is a summary of the call. Specifically, the server uses an alert message generation means to create a summary such as, "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery."

[0791] Step 6:

[0792] The server sends the generated alert message to the person and their emergency contact. The input is a summary of the call, and the output is the alert message. Specifically, the server uses the notification method to send a message via a push notification service saying, "There was a call to your mother's phone that is likely fraudulent. Please check it immediately."

[0793] Step 7:

[0794] The device receives an alert message sent from the server and notifies the user via a push notification. The input is the alert message from the server, and the output is a notification displayed on the user's device screen. Specifically, the device displays the received notification as a pop-up on the screen.

[0795] Step 8:

[0796] The user sees the alert message displayed on the device. The input is a summary of the call displayed on the device, and the output provides information that the user can use to determine the possibility of fraud. The user opens the app and sees the detailed summary of the call.

[0797] Step 9:

[0798] If the user determines that there is a high possibility of fraud, they will end the call and report it to the police or emergency contacts. The input is the user's fraud judgment, and the output is a report. Specifically, the user will end the call promptly and contact the police or emergency contacts.

[0799] (Application example 1)

[0800] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0801] Conventional telephone fraud prevention systems analyze voice data to determine the possibility of fraud, but have the problem that users cannot receive visual warnings of fraud while on the phone. Also, if the smart device owner is physically away from the phone, it is difficult to respond quickly. This has led to the problem that fraud victims cannot be sufficiently prevented from falling victim to fraud.

[0802] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0803] In this invention, the server includes means for analyzing voice data received from a telephone terminal, means for converting the voice data into text, means for determining the possibility of special fraud from the text data, means for generating an alert message based on the determination, means for sending the alert message to the user and an emergency contact, and means for displaying a visual alert on a smart device. This allows the user to instantly receive a visual fraud warning during a call, enabling a prompt response even when the user is physically away from the telephone.

[0804] A "telephone terminal" is a communication device for making voice calls.

[0805] "Audio data" refers to data that records or transmits audio signals in digital form.

[0806] A "server" is a computer system for processing and managing data.

[0807] "Speech recognition means" is a technology that converts voice data into text data.

[0808] "Text data" is data expressed as a string of characters.

[0809] "Analysis means" refers to the technology that analyzes given data and extracts specific information or patterns.

[0810] An "alert message" is a message that contains warning or notification content.

[0811] A "notification means" is a technique for conveying information to a particular device or user.

[0812] A "smart device" is a highly functional electronic device equipped with communication and computer functions.

[0813] A "visual alert" is a method of displaying warnings or notifications using visual elements.

[0814] MODE FOR CARRYING OUT THE INVENTION

[0815] The present invention provides a system that analyzes calls received from a telephone terminal in real time and issues an alert if a special fraud is suspected. The following describes in detail the embodiments of the present invention.

[0816] Server-side implementation

[0817] The server receives voice data from the telephone terminal in real time. The voice data is converted into text data by a voice recognition means. For example, if a caller says, "Your son was in a traffic accident and urgently needs surgery," the call is recorded as text: "Your son was in a traffic accident and urgently needs surgery." The analysis means then analyzes the text data to determine whether it is likely fraudulent. In this case, keywords such as "traffic accident" and "surgery costs" are detected, indicating a high likelihood of fraud. The server then uses an alert message generation means to create a summary of the potentially fraudulent call. For example, the summary might read, "This call is about someone claiming to be your son being in a traffic accident and needing surgery." The notification means then sends an alert message to the caller and their emergency contact. Specifically, the message might read, "There has been a potentially fraudulent call to your mother's phone. Please check immediately." A visual alert display on the smart device is also provided. This allows for a prompt response even when the caller is away from the phone.

[0818] Terminal side embodiment

[0819] When a device receives a call from a number not registered in the phone book, it automatically forwards the call to the server. At this time, it appears to the user as if the call were normal, but since analysis is performed on the server, user operation is minimal. In addition, when a notification from the server arrives on the device or smart device, the device notifies the user of the content via push notification. Upon receiving the notification, the user can check a summary of the call content.

[0820] User's embodiment

[0821] When a user receives a notification from the server during a call, they open the application and check a summary of the call. For example, they might see something like, "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely fraudulent, they can quickly end the call and report it to the police or emergency contacts. A visual alert displayed on the smart device also allows for quick action.

[0822] Hardware and Software

[0823] Hardware:

[0824] server

[0825] telephone terminal

[0826] Smart devices (e.g., smart glasses, smartphones)

[0827] software:

[0828] Speech recognition: speech_recognition

[0829] Translation: googletrans

[0830] Text analysis: textblob

[0831] Specific examples

[0832] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for surgery costs. When this call begins, the device automatically transfers the call contents to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If the server determines that the call is likely to be fraudulent, it generates an alert message and sends it to Person A and his / her emergency contacts. Person A receives the notification during the call, checks the content, and determines that it is fraudulent. By ending the call and reporting it to the police, the fraud can be prevented. In addition, a visual alert is displayed on smart devices such as smart glasses, so the warning can be received immediately even if the person is away from the phone.

[0833] Prompt Sentence Examples

[0834] If a user wants to add fraud keywords to a generative AI model, here is an example prompt:

[0835] "Add new scam keyword: {new keyword}"

[0836] for example:

[0837] "Add new fraud keywords: bank account, PIN, payment due date"

[0838] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0839] Step 1:

[0840] The terminal detects that there is an incoming call to the telephone terminal. If the call is from a number not registered in the phone book, the terminal transfers the call to the server, which then sends the voice data as input data to be provided to the server.

[0841] Step 2:

[0842] The server converts the received voice data into text data in real time using a speech recognition tool. This conversion process uses speech recognition software (speech_recognition). The input is voice data, and the output is text data that expresses the voice in characters.

[0843] Step 3:

[0844] The server analyzes the converted text data using analytical means to determine the possibility of special fraud. This analysis uses a text analysis library (textblob) and a translation library (googletrans). The input is text data, and data calculations are performed to detect fraud keywords and evaluate their likelihood. The output is an evaluation result indicating the possibility of fraud.

[0845] Step 4:

[0846] If the server determines that fraud is likely, it generates an alert message using an alert message generation means. Specifically, it generates a text message summarizing the details of the fraud. The input is the evaluation result and the text data that may be fraudulent, and the output is the alert message.

[0847] Step 5:

[0848] The server sends the generated alert message to the individual and their emergency contacts using a notification mechanism. Additionally, a visual alert is displayed on the smart device (e.g., smart glasses, smartphone). The notification mechanism uses a push notification service. The input is the alert message, and the output is the notification transmission result.

[0849] Step 6:

[0850] The user receives a push notification from their terminal or smart device during a call and checks the content. The user opens the application and reads a summary of the call to determine whether it is fraudulent. The input is the alert message, and the output is the user's decision.

[0851] Step 7:

[0852] If the user determines that there is a high possibility of fraud, they can quickly end the call and report it to the police or an emergency contact. This makes it possible to prevent actual fraud from occurring. The input is the user's judgment, and the output is the reporting action.

[0853] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0854] The present invention relates to a system that analyzes calls received from telephone terminals in real time and issues an alert if a special fraud is suspected. In particular, the present invention improves the accuracy of fraud detection by combining voice data analysis with an emotion engine that recognizes the user's emotions.

[0855] Server-side implementation

[0856] The server receives voice data sent from the terminal in real time. The voice data is converted into text data by a voice recognition means. For example, if someone says over the phone, "Your son has been in a traffic accident and needs urgent surgery," the content is recorded as text: "Your son has been in a traffic accident and needs urgent surgery."

[0857] Next, an analysis means analyzes this text data to determine whether it is likely to be fraud. In this case, keywords such as "traffic accident" and "surgery costs" are detected, and it is determined that there is a high possibility of fraud.

[0858] Additionally, the server is equipped with an emotion engine that analyzes the user's emotional state from the voice data. The emotion engine detects emotions such as stress or anxiety, and provides this to the analysis means as auxiliary information to further strengthen suspicion of fraud.

[0859] The server then uses an alert message generator to create a summary of the potentially fraudulent call. For example, it could summarize the call as "This call is about a person claiming to be your son who was in a car accident and needs money for surgery." An alert message is then sent to the person and their emergency contacts via a notification mechanism. Specifically, the message could read, "Mom's phone has received a potentially fraudulent call. Please check immediately."

[0860] Terminal side embodiment

[0861] When a call comes in from a number not registered in the phone book, the device automatically forwards the call to the server. At this time, it appears to the user as if the call is normal, but since analysis is done on the server, user operation is minimal.

[0862] When a notification from the server arrives at the device, the device notifies the user of the content via push notification. By receiving the notification, the user can check a summary of the call content.

[0863] User's embodiment

[0864] If a user receives a notification from the server during a call, they can open the app and check a summary of the call. For example, they might see something like, "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely to be fraud, they can quickly end the call and report it to the police or an emergency contact.

[0865] Specific examples

[0866] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, the device automatically transfers the contents of the call to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If the server determines that there is a high possibility of fraud, it uses an emotion engine to analyze the user's emotional state, and if suspicious emotions are confirmed, it incorporates this as supplementary information. Finally, an alert message is generated and sent to Person A and their emergency contact. After Person A receives the notification during the call and checks the content, he or she determines that it is a fraud, ends the call, and reports it to the police, thereby preventing fraud from occurring.

[0867] As described above, the present invention provides a specific method and system for detecting special telephone frauds in real time and responding quickly by making full use of an emotion engine.

[0868] The processing flow will be explained below.

[0869] Server-side processing steps

[0870] Step 1:

[0871] The server receives a call start signal from the terminal. When the terminal detects a call from an unregistered number and begins preparations to transfer the call to the server, a call start notification is sent to the server.

[0872] Step 2:

[0873] The server receives the voice data in real time. When a call is started, the device sends the voice data in streaming format to the server, which stores it in a buffer.

[0874] Step 3:

[0875] The server activates the speech recognition means and converts the received voice data into text. For example, the contents of the call may be converted into text such as "Your son was in a traffic accident and needs urgent medical expenses."

[0876] Step 4:

[0877] The server uses an analytical method to analyze the text data, detect fraud-related keywords (e.g., "traffic accident," "surgery costs," etc.), and determine whether there is any suspicion of fraud.

[0878] Step 5:

[0879] The server activates an emotion engine to analyze the user's emotional state (e.g., anxiety, stress, etc.) from the received voice data. This emotional state is treated as auxiliary information to strengthen suspicion of fraud.

[0880] Step 6:

[0881] The server generates an alert message based on the text data and the results of sentiment analysis. For example, a summary such as "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery" is generated.

[0882] Step 7:

[0883] The server will then use the notification method to send an alert message to the individual and their emergency contacts, such as "There was a potentially fraudulent call to your mother's phone. Please check immediately."

[0884] Terminal processing steps

[0885] Step 1:

[0886] The device receives a call from a number that is not registered in the phone book. When a call comes in from an unregistered number, the call is automatically forwarded to the server. When the call starts, the voice data is sent to the server.

[0887] Step 2:

[0888] The device sends call data to the server in real time. The call audio is recorded on the device and transferred to the server in streaming format.

[0889] Step 3:

[0890] The device waits for and receives alert notifications from the server. If the server determines that fraud is suspected and generates an alert, the notification is sent to the device.

[0891] Step 4:

[0892] The device will display an alert message to the user. When a push notification is received, the notification will say, "Warning: There has been a potentially fraudulent call. Please check the details."

[0893] User processing steps

[0894] Step 1:

[0895] The user receives a notification on the device. When an alert notification arrives on the device during a call, it is visible to the user.

[0896] Step 2:

[0897] The user opens the app and checks the summary of the call. The summary displayed in the app reads, "This call is from someone claiming to be your son and he needs money for surgery after a traffic accident."

[0898] Step 3:

[0899] The user should end the call and, if necessary, contact the police or emergency contacts. If there is a strong suspicion of fraud, the user should immediately end the call and take appropriate action, such as reporting the call to the police.

[0900] In this way, a system is realized in which the server, terminal, and user work together to reduce the risk of special fraud. By combining this system with an emotion engine, the accuracy of fraud detection can be improved, enabling more effective fraud countermeasures.

[0901] Example 2

[0902] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0903] Special fraud methods are becoming more sophisticated every year, and it is extremely difficult for victims to recognize fraud in real time, especially in cases of fraud over the phone. As a result, many people fall victim to fraud and suffer huge losses. In addition, it is necessary to accurately analyze the content of fraudulent calls and detect the emotional state of the victim, but conventional systems are inadequate in dealing with this important aspect.

[0904] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0905] In this invention, the server includes means for analyzing voice data received from a telephone terminal, means for converting the voice data into text, means for determining the possibility of special fraud from the text data, means for analyzing the emotional state using the analysis means, means for generating an alert message based on the determination, and means for sending the alert message to the caller and their emergency contacts. This makes it possible to analyze the content of fraudulent calls in real time and quickly issue an alert if there is a high possibility of fraud. Furthermore, by analyzing the victim's emotional state, the accuracy of fraud detection can be improved, helping to prevent damage before it occurs.

[0906] "Server" refers to a computer system that analyzes and manages voice data received from telephone terminals, and determines and notifies users of the possibility of fraud.

[0907] "Voice data" refers to the voice signal received from the telephone terminal, which is the subject of analysis.

[0908] "Speech recognition means" refers to the process or technology that converts voice data into text data, and a speech recognition engine is an example of this.

[0909] "Text data" refers to data in sentence format converted by a speech recognition means, and serves as the basis for analysis.

[0910] "Analysis methods" refers to the processes and technologies that analyze text data, detect specific keywords and phrases, and determine the possibility of special fraud.

[0911] "Emotion analysis means" refers to processes or technologies that analyze a user's emotional state from voice or text data and provide supplemental information that reinforces the possibility of fraud.

[0912] "Alert Message" refers to a warning message summarizing the content of a call that is determined to be highly likely to be fraudulent.

[0913] "Message generation means" refers to the process or technology for creating an alert message based on the results of the analysis means and the sentiment analysis means.

[0914] "Means of Notification" refers to the process or technology by which the generated alert message is sent to the individual and their emergency contacts.

[0915] "Push notification service" refers to a service that sends messages from a server to a user's device in real time.

[0916] "Telephone terminal" refers to a device for making voice calls.

[0917] This invention relates to a system that analyzes calls received from telephone terminals in real time and issues an alert if a special fraud is suspected. Specifically, it improves the accuracy of fraud detection by combining voice data analysis with an emotion analysis engine that recognizes user emotions.

[0918] Server-side implementation

[0919] The server receives voice data sent from the telephone terminal in real time. This voice data is converted into text data using a speech recognition means. For example, the voice data is converted into text data using the Google Cloud Speech-to-Text API. As a specific example, if someone says over the phone, "Your son has been in a traffic accident and needs urgent medical expenses," the content of the call is recorded as text as "Your son has been in a traffic accident and needs urgent medical expenses."

[0920] The server then uses a natural language processing engine, such as IBM Watson Natural Language Understanding, to analyze the text data. This process determines whether there is a possibility of fraud. For example, if certain keywords, such as "traffic accident" or "surgery costs," are detected, it is determined that there is a high possibility of fraud.

[0921] The server is also equipped with an emotion analysis engine that analyzes the user's emotional state from the voice data. For example, Emotion AI is used to detect emotions such as stress and anxiety. This emotional data is provided to the analysis means as auxiliary information to further increase the likelihood of fraud.

[0922] Furthermore, the server generates an alert message based on the result of the judgment. The alert message generation means creates a summary such as, "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery." The notification means then sends the alert message to the person and their emergency contact. Specifically, the message sent may read, "There has been a call to your mother's phone that is likely to be fraudulent. Please check immediately."

[0923] Terminal side embodiment

[0924] When a call comes in from a number not registered in the phone book, the device automatically forwards the call to the server. At this time, it appears to the user as if the call is normal, but since analysis is done on the server, user operation is minimal.

[0925] In addition, when a notification from the server arrives on the device, the device will notify the user of the content via push notification. When the notification is received, the user can easily check the content of the notification. For example, a prompt message may be sent saying, "There was a call to your mother's phone that is likely to be fraudulent. Please check it immediately."

[0926] User's embodiment

[0927] When a user receives a notification from the server during a call, they can open a dedicated app and check a summary of the call. For example, a summary such as "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery" may be displayed. If the user determines based on this summary that the call is likely to be fraud, they can immediately end the call and report it to the police or an emergency contact.

[0928] Specific examples

[0929] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, Person A's device automatically transfers the voice call to the server. The server receives the voice data and converts it into text using the Google Cloud Speech-to-Text API. It then analyzes the text data using IBM Watson Natural Language Understanding to detect keywords such as "traffic accident" and "surgery costs." If the server determines that there is a high possibility of fraud, it uses Emotion AI to analyze the user's emotional state, and if emotions such as stress or anxiety are detected, it incorporates this as supplemental information.

[0930] Finally, the server generates an alert message and sends it to Person A and his / her emergency contacts. After Person A receives the notification during the call and checks the summary, he / she determines that the call is likely a scam, immediately ends the call, and reports it to the police. This prevents the scam from occurring.

[0931] As described above, the present invention provides a specific method and system for detecting special telephone frauds in real time and responding quickly by making full use of an emotion analysis engine.

[0932] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0933] Step 1:

[0934] When a terminal receives a call from a number not registered in the phone book, it automatically transfers the call contents to the server. It receives voice data from the telephone terminal as input and sends this voice data to the server as output. From the user's side, it appears as if a normal call is being made, but in reality, the call contents are being sent to the server in real time.

[0935] Step 2:

[0936] The server receives voice data sent from the terminal in real time. It receives voice data from the terminal as input and passes it to the voice recognition means as output. Specifically, the voice data is sent to the server in streaming format, and the server supplies it to the voice recognition engine.

[0937] Step 3:

[0938] The server converts the received voice data into text data using a speech recognition method (for example, Google Cloud Speech-to-Text API). As input, it receives real-time voice data and converts it into text data. The output is the converted text data. For example, a message such as "Your son has been in a traffic accident and urgently needs money for surgery" may be recorded as text.

[0939] Step 4:

[0940] The server analyzes the text data using analytical means (for example, IBM Watson Natural Language Understanding). It receives the text data as input and detects specific keywords or phrases contained within it. The output is a determination result as to whether or not the keyword or phrase is included. For example, if keywords such as "traffic accident" or "surgery costs" are detected, it determines that the call is likely to be fraudulent.

[0941] Step 5:

[0942] The server uses emotion analysis means (e.g., Emotion AI) to analyze the user's emotional state from the voice data. As input, it receives the original voice data and text data and detects emotions such as stress and anxiety. The output is the detected emotion data. For example, if high anxiety or stress is detected, this is used as auxiliary information to strengthen the possibility of fraud.

[0943] Step 6:

[0944] The server generates an alert message based on the analysis results and sentiment analysis results. It receives the text analysis results and sentiment analysis data as input, and creates a summarized warning message using the alert message generation means. For example, it generates a summary such as "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery." The output is this summary message.

[0945] Step 7:

[0946] The server sends the generated alert message to the user and emergency contacts using a notification method. It receives the generated alert message as input and sends it as output to the user and emergency contacts via email, SMS, or push notification services. Specifically, it sends a message saying, "There was a likely fraudulent call on your mother's phone. Please check it immediately."

[0947] Step 8:

[0948] If a user receives an alert notification during a call, the user opens a dedicated app and checks a summary of the call. The input is receiving the alert notification and displaying the notification content within the application. The output is summary information that is displayed on the user's screen. For example, it might say, "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery."

[0949] Step 9:

[0950] If the user determines based on this summary information that the call is likely to be fraudulent, they can immediately end the call and report it to the police or emergency contacts. The input is to check the alert notification and summary information and take appropriate action. The output is to end the call and report it to the police. For example, a fraudulent call can be reported using the "Report" button provided in the dedicated app.

[0951] (Application example 2)

[0952] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0953] The methods of special frauds using telephone terminals are becoming more sophisticated, and existing systems may have difficulty detecting them. Furthermore, systems that ignore the user's emotional state have the problem of being unable to adequately determine the possibility of fraud. Therefore, there is a need for a system that can detect fraud with greater accuracy by analyzing both voice data and the user's emotional state.

[0954] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0955] In this invention, the server includes means for analyzing voice data received from a telephone terminal, means for converting the voice data into text, means for determining the possibility of special fraud from the text data, means for generating an alert message, means for sending the alert message to the user and an emergency contact, emotion analysis means for detecting the user's emotional state from the voice data, and means for using the output of the emotion analysis means as supplementary information for the determination. This makes it possible to analyze both the voice data and the user's emotional state and detect the possibility of fraud with higher accuracy.

[0956] A "server" is a central computer system that analyzes voice data received from telephone terminals and determines the possibility of fraud.

[0957] "Speech recognition means" refers to a technique or device for converting received voice data into text data.

[0958] "Analysis means" refers to a technology or device for analyzing text data to determine the possibility of special fraud.

[0959] The "alert message generating means" is a technology or device that, when the analysis means determines that there is a possibility of fraud, generates a message to notify the user or emergency contact based on that information.

[0960] "Notification means" refers to the technology or device used to send the generated alert message to the individual and their emergency contacts.

[0961] An "emotion analysis means" is a technique or device for detecting a user's emotional state (for example, stress or anxiety) from voice data.

[0962] "Supplementary information" is information for incorporating data on the user's emotional state obtained by the emotion analysis means into the determination process of the analysis means.

[0963] The present invention provides a detailed description of a system that detects special frauds in real time via telephone terminals and sends alerts to users and emergency contacts.

[0964] Server embodiment

[0965] The server receives voice data sent from the telephone terminal in real time. The hardware required is a server machine with a high-performance processor and sufficient memory, and the software uses the "Google Speech Recognition API" as the voice recognition engine, the "EmotionalEngine" as the emotion analysis engine, and the "AlertSystem" as the alert system. The server's processing procedure is as follows:

[0966] First, the server receives the voice data and converts it into text data using a voice recognition device. For example, if the caller says, "My son was in a traffic accident and needs urgent surgery money," the content of the call will be recorded as text data.

[0967] Next, an analysis means analyzes this text data to determine the possibility of fraud. At this stage, specific keywords such as "traffic accident" or "surgery costs" are detected. The server also has an emotion analysis means that detects the user's emotional state (e.g., stress or anxiety) from the voice data. This information is provided to the analysis means as supplementary information to further strengthen suspicion of fraud.

[0968] Based on the analysis results, the alert message generating means creates a summary of the potentially fraudulent call, and the notification means sends an alert message to the user and emergency contacts. Specifically, a message such as "There has been a potentially fraudulent call to your mother's phone. Please check it immediately" is sent.

[0969] Terminal embodiment

[0970] The device has a function that automatically forwards calls from numbers not registered in the phone book to the server. At this time, it appears to the user as if they are making a normal call, but the content of the call is analyzed by the server, so the user's operation is minimal.

[0971] When a notification from the server arrives at the device, the device notifies the user of the content via push notification. By receiving the notification, the user can check a summary of the call content.

[0972] User's embodiment

[0973] If a user receives a notification from the server during a call, they can open the "Fraud Guard" application installed on their smartphone and check a summary of the call. For example, it might say, "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely to be fraud, they can quickly end the call and report it to the police or an emergency contact.

[0974] Specific examples

[0975] As a concrete example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, the device automatically transfers the contents of the call to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If it determines that there is a high possibility of fraud, the server performs an emotion analysis, and if it confirms that the user is in a state of high stress, it incorporates this as supplementary information. Finally, an alert message is generated and sent to Person A and their emergency contacts.

[0976] Prompt Sentence Examples

[0977] Example prompts to input to a generative AI model:

[0978] Perform voice analysis, and if the converted text contains words like "traffic accident" or "surgery costs," use emotion analysis to check for stress or anxiety, and generate a program that sends an alert to the user if there is a suspicion of fraud.

[0979] As described above, the present invention provides a specific method and system for detecting special telephone frauds in real time and responding quickly by making full use of an emotion engine.

[0980] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0981] Step 1:

[0982] The telephone terminal receives an incoming call.

[0983] Input: Audio data when a phone call is received.

[0984] Specific operation: When a call is received from an unregistered number, the device starts the call and transfers the voice data to the server in real time.

[0985] Output: Audio data transferred to the server.

[0986] Step 2:

[0987] The server receives the audio data.

[0988] Input: Audio data transferred from the device.

[0989] Specific operation: The server receives audio data in real time using a high-performance data stream processing device.

[0990] Output: The audio data is saved to the server's storage device.

[0991] Step 3:

[0992] The server converts the voice data into text data.

[0993] Input: Stored audio data.

[0994] What it does: Uses the Google Speech Recognition API to convert audio data into corresponding text data.

[0995] Output: Text data.

[0996] Step 4:

[0997] The server analyzes the text data and determines whether it is a special fraud.

[0998] Input: The converted text data.

[0999] Specific operation: Searches for specific keywords (e.g., "traffic accident" or "surgery costs") in the text data. If the conditions are met, it is determined to be a possible fraud.

[1000] Output: Determination result regarding likelihood of fraud.

[1001] Step 5:

[1002] The server detects the user's emotional state from the voice data.

[1003] Input: Received audio data.

[1004] Specific behavior: Uses EmotionalEngine to detect the user's emotional state (e.g., stress or anxiety) from audio data.

[1005] Output: Detected user emotional state data.

[1006] Step 6:

[1007] The server uses the emotional state data as supplementary information for analysis.

[1008] Input: Emotional state data and analysis results.

[1009] What it does: If the user is in a state of high stress or anxiety, that information is used to supplement the analysis results and make a stronger judgment about the likelihood of fraud.

[1010] Output: The final fraud verdict taking into account the supplementary information.

[1011] Step 7:

[1012] The server generates an alert message.

[1013] Input: Final fraud determination result.

[1014] Specific action: Based on the judgment result, a specific alert message (e.g., "This call may be fraudulent. Please check the content.") is automatically generated.

[1015] Output: The alert message.

[1016] Step 8:

[1017] The server sends an alert message via a notification means.

[1018] Input: The generated alert message.

[1019] Specific operation: Uses push notification service to send an alert message to the user and their emergency contacts.

[1020] Output: Sending an alert message to the user and emergency contacts.

[1021] Step 9:

[1022] The user acknowledges the alert message.

[1023] Input: The received alert message.

[1024] Specific actions: Open the Fraud Guard app installed on your smartphone and check the call summary and alert message.

[1025] Output: The user becomes aware of the possibility of fraud and takes necessary measures.

[1026] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1027] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1028] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1029] [Fourth embodiment]

[1030] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1031] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1032] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1033] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1034] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1035] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1036] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1037] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1038] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1039] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1040] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1041] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1042] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1043] The present invention relates to a system that analyzes calls received from telephone terminals in real time and issues an alert if a special fraud is suspected. This system works in cooperation with a server, a terminal, and a user.

[1044] Server-side implementation

[1045] The server receives voice data sent from the terminal in real time. The voice data is converted into text data by a voice recognition means. For example, if someone says over the phone, "Your son has been in a traffic accident and needs urgent surgery," the content is recorded as text: "Your son has been in a traffic accident and needs urgent surgery."

[1046] Next, an analysis means analyzes this text data to determine whether it is likely to be fraud. In this case, keywords such as "traffic accident" and "surgery costs" are detected, and it is determined that there is a high possibility of fraud.

[1047] The server then uses an alert message generator to create a summary of the potentially fraudulent call. For example, it could summarize the call as "This call is about a person claiming to be your son who was in a car accident and needs money for surgery." An alert message is then sent to the person and their emergency contacts via a notification mechanism. Specifically, the message could read, "Mom's phone has received a potentially fraudulent call. Please check immediately."

[1048] Terminal side embodiment

[1049] When a call comes in from a number not registered in the phone book, the device automatically forwards the call to the server. At this time, it appears to the user as if the call is normal, but since analysis is done on the server, user operation is minimal.

[1050] When a notification from the server arrives at the device, the device notifies the user of the content via push notification. By receiving the notification, the user can check a summary of the call content.

[1051] User's embodiment

[1052] If a user receives a notification from the server during a call, they can open the app and check a summary of the call. For example, they might see something like, "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely to be fraud, they can quickly end the call and report it to the police or an emergency contact.

[1053] Specific examples

[1054] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, the device automatically transfers the contents of the call to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If the server determines that there is a high possibility of fraud, it generates an alert message and sends it to Person A and his emergency contacts. Person A receives the notification during the call, checks the content, and determines that it is a fraud. By ending the call and reporting it to the police, the fraud can be prevented.

[1055] As described above, the present invention provides a specific method and system for detecting special telephone frauds in real time and responding quickly.

[1056] The processing flow will be explained below.

[1057] Server-side processing steps

[1058] Step 1:

[1059] The server receives a call start signal from the terminal. When the terminal detects a call from an unregistered number and begins preparations to transfer the call to the server, a call start notification is sent to the server.

[1060] Step 2:

[1061] The server receives the voice data in real time. When a call is started, the device sends the voice data in streaming format to the server, which stores it in a buffer.

[1062] Step 3:

[1063] The server activates the speech recognition means and converts the received speech data into text. For example, "Your son was in a traffic accident and needs urgent medical expenses."

[1064] Step 4:

[1065] The server uses analytical means to analyze the text data and detects specific fraud-related keywords (e.g., "traffic accident," "surgery costs," etc.). If such keywords are detected, it is determined that there is a suspicion of special fraud.

[1066] Step 5:

[1067] The server generates an alert message based on the text data, such as a summary of the call, such as "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery."

[1068] Step 6:

[1069] The server uses a notification method to send an alert message to the individual and their emergency contacts, such as "There was a potentially fraudulent call to your mother's phone. Please check it immediately."

[1070] Terminal processing steps

[1071] Step 1:

[1072] The device receives a call from a number that is not registered in the phone book. When a call comes in from an unregistered number, the call is automatically forwarded to the server.

[1073] Step 2:

[1074] The device sends call data to the server in real time. The call audio is recorded on the device and transferred to the server in streaming format.

[1075] Step 3:

[1076] The device waits for and receives alert notifications from the server. If the server determines that fraud is suspected and generates an alert, the notification is sent to the device.

[1077] Step 4:

[1078] The device will display an alert message to the user. When a push notification is received, the notification will say, "Warning: There has been a potentially fraudulent call. Please check the details."

[1079] User processing steps

[1080] Step 1:

[1081] The user receives a notification on the device. When an alert notification arrives on the device during a call, it is visible to the user.

[1082] Step 2:

[1083] The user opens the app and checks the summary of the call. From the summary displayed in the app, the user understands that "This call is about a person claiming to be my son who was in a traffic accident and needs money for surgery."

[1084] Step 3:

[1085] The user should end the call and, if necessary, contact the police or emergency contacts. If the user suspects fraud, they should immediately end the call and take appropriate action, such as reporting it to the police.

[1086] In this way, a system is realized in which the server, terminal, and user work together to reduce the risk of special fraud.

[1087] Example 1

[1088] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1089] In recent years, the number of victims of special telephone fraud has been increasing, with many elderly people falling victim to these frauds. To solve this problem, a system is needed that monitors the content of phone calls in real time and immediately issues an alert if it is determined that there is a high possibility of fraud. However, currently, there is no adequate system in place to achieve this kind of real-time analysis, making it difficult to take effective countermeasures. Therefore, the objective of this invention is to provide a system that automatically analyzes the content of phone calls and quickly issues an alert if fraud is suspected.

[1090] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1091] In this invention, the server includes means for analyzing voice data received from a telephone terminal in real time, means for converting the voice data into text, means for determining the possibility of special fraud from the text data, means for generating a summary of the call based on the determination, and means for sending the summary of the call to the person in question and an emergency contact, thereby making it possible to detect the possibility of fraud during a call in real time and to notify the person promptly.

[1092] A "server" is a computer system that receives the voice data, analyzes it, and processes any necessary notifications.

[1093] A "telephone terminal" is a communication device for making voice calls, and has the function of transferring voice data to a server.

[1094] "Voice data" refers to digital voice information of the contents of a call received from a telephone terminal.

[1095] "Speech recognition means" refers to technology or software for converting voice data into text data.

[1096] "Text data" is character string information generated from voice data by voice recognition means.

[1097] "Analysis means" refers to technology or software used to analyze text data and determine the possibility of fraud.

[1098] An "alert message" is a notification containing warning information about a call that is likely to be fraudulent.

[1099] "Notification Method" refers to the technology or software used to send an alert message to you and your emergency contacts.

[1100] A "call summary" is text information that briefly summarizes the contents of a call that is likely to be fraudulent.

[1101] "Push notification service" is a communication technology for sending messages from a server to a device in real time.

[1102] MODE FOR CARRYING OUT THE INVENTION

[1103] The present invention relates to a system that analyzes calls received from telephone terminals in real time and issues an alert if a special fraud is suspected. This system functions in cooperation with a server, telephone terminals, and users.

[1104] Server-side implementation

[1105] The server first receives voice data sent from the telephone terminal in real time. The voice data is converted into text data using a speech recognition method such as Google Cloud Speech-to-Text API. For example, if a caller says, "Your son was in a traffic accident and needs urgent surgery," the server converts the content into text that reads, "Your son was in a traffic accident and needs urgent surgery."

[1106] Next, the server analyzes the text data using Python's natural language processing libraries, NLTK and SpaCy. This analysis detects keywords such as "traffic accident" and "surgery costs," and determines whether there is a suspicion of fraud. In this case, it is determined that there is a high suspicion of fraud.

[1107] The server then uses an alert message generator to create a summary of the potentially fraudulent call. For example, it could summarize the call as "This call is about someone claiming to be your son who was in a car accident and needs money for surgery." The server then uses a notification mechanism to send an alert message to the person and their emergency contact. Specifically, the message could read, "Mom's phone has received a potentially fraudulent call. Please check immediately."

[1108] Terminal side embodiment

[1109] When a phone device receives a call from a number not registered in the phone book, it automatically forwards the call to the server. Although it appears to the user as if the call is normal, analysis is performed on the server, so the user's operation is minimal. In addition, when a notification from the server arrives on the device, the device notifies the user of the content via a push notification. Upon receiving the notification, the user can check a summary of the call content.

[1110] User's embodiment

[1111] If a user receives a notification from the server during a call, they can open the app and check a summary of the call. For example, they might see something like, "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely to be fraud, they can quickly end the call and report it to the police or an emergency contact.

[1112] Specific examples

[1113] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, the telephone terminal automatically transfers the contents of the call to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If the server determines that there is a high possibility of fraud, it generates an alert message and sends it to Person A and his emergency contacts. Person A receives the notification during the call, checks the content, and determines that it is a fraud. By ending the call and reporting it to the police, the fraud can be prevented.

[1114] Prompt Sentence Examples

[1115] "Please tell me how this system can be used to detect special telephone fraud in real time. For example, please explain in detail how you would respond if someone said, 'My son was in a traffic accident.'"

[1116] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1117] Step 1:

[1118] The device receives a call from a number that is not registered in the phone book. The input is the incoming phone number, and the output is that the call from that phone number is automatically recorded and forwarded to the server. Specifically, the device records the contents of the call in real time and sends the recorded data to the server.

[1119] Step 2:

[1120] The server receives the voice data transferred from the terminal in real time. The input is the voice data sent from the terminal, and the output is the voice data stored on the server. Specifically, the server stores the voice data in an appropriate format.

[1121] Step 3:

[1122] The server converts the voice data into text data using the Google Cloud Speech-to-Text API. The input is the stored voice data, and the output is the generated text data. Specifically, the server sends the voice data to the speech recognition API and saves the received text data.

[1123] Step 4:

[1124] The server analyzes the text data to determine whether it is likely to be fraud. The input is the generated text data, and the output is the analysis result. Specifically, the server uses Python's NLTK and SpaCy to extract keywords such as "traffic accident" and "surgery costs" from the text data, and determines the possibility of fraud based on these keywords.

[1125] Step 5:

[1126] The server generates a summary of the call if there is a high possibility of fraud. The inputs are the analysis results and text data, and the output is a summary of the call. Specifically, the server uses an alert message generation means to create a summary such as, "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery."

[1127] Step 6:

[1128] The server sends the generated alert message to the person and their emergency contact. The input is a summary of the call, and the output is the alert message. Specifically, the server uses the notification method to send a message via a push notification service saying, "There was a call to your mother's phone that is likely fraudulent. Please check it immediately."

[1129] Step 7:

[1130] The device receives an alert message sent from the server and notifies the user via a push notification. The input is the alert message from the server, and the output is a notification displayed on the user's device screen. Specifically, the device displays the received notification as a pop-up on the screen.

[1131] Step 8:

[1132] The user sees the alert message displayed on the device. The input is a summary of the call displayed on the device, and the output provides information that the user can use to determine the possibility of fraud. The user opens the app and sees the detailed summary of the call.

[1133] Step 9:

[1134] If the user determines that there is a high possibility of fraud, they will end the call and report it to the police or emergency contacts. The input is the user's fraud judgment, and the output is a report. Specifically, the user will end the call promptly and contact the police or emergency contacts.

[1135] (Application example 1)

[1136] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1137] Conventional telephone fraud prevention systems analyze voice data to determine the possibility of fraud, but have the problem that users cannot receive visual warnings of fraud while on the phone. Also, if the smart device owner is physically away from the phone, it is difficult to respond quickly. This has led to the problem that fraud victims cannot be sufficiently prevented from falling victim to fraud.

[1138] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1139] In this invention, the server includes means for analyzing voice data received from a telephone terminal, means for converting the voice data into text, means for determining the possibility of special fraud from the text data, means for generating an alert message based on the determination, means for sending the alert message to the user and an emergency contact, and means for displaying a visual alert on a smart device. This allows the user to instantly receive a visual fraud warning during a call, enabling a prompt response even when the user is physically away from the telephone.

[1140] A "telephone terminal" is a communication device for making voice calls.

[1141] "Audio data" refers to data that records or transmits audio signals in digital form.

[1142] A "server" is a computer system for processing and managing data.

[1143] "Speech recognition means" is a technology that converts voice data into text data.

[1144] "Text data" is data expressed as a string of characters.

[1145] "Analysis means" refers to the technology that analyzes given data and extracts specific information or patterns.

[1146] An "alert message" is a message that contains warning or notification content.

[1147] A "notification means" is a technique for conveying information to a particular device or user.

[1148] A "smart device" is a highly functional electronic device equipped with communication and computer functions.

[1149] A "visual alert" is a method of displaying warnings or notifications using visual elements.

[1150] MODE FOR CARRYING OUT THE INVENTION

[1151] The present invention provides a system that analyzes calls received from a telephone terminal in real time and issues an alert if a special fraud is suspected. The following describes in detail the embodiments of the present invention.

[1152] Server-side implementation

[1153] The server receives voice data from the telephone terminal in real time. The voice data is converted into text data by a voice recognition means. For example, if a caller says, "Your son was in a traffic accident and urgently needs surgery," the call is recorded as text: "Your son was in a traffic accident and urgently needs surgery." The analysis means then analyzes the text data to determine whether it is likely fraudulent. In this case, keywords such as "traffic accident" and "surgery costs" are detected, indicating a high likelihood of fraud. The server then uses an alert message generation means to create a summary of the potentially fraudulent call. For example, the summary might read, "This call is about someone claiming to be your son being in a traffic accident and needing surgery." The notification means then sends an alert message to the caller and their emergency contact. Specifically, the message might read, "There has been a potentially fraudulent call to your mother's phone. Please check immediately." A visual alert display on the smart device is also provided. This allows for a prompt response even when the caller is away from the phone.

[1154] Terminal side embodiment

[1155] When a device receives a call from a number not registered in the phone book, it automatically forwards the call to the server. At this time, it appears to the user as if the call were normal, but since analysis is performed on the server, user operation is minimal. In addition, when a notification from the server arrives on the device or smart device, the device notifies the user of the content via push notification. Upon receiving the notification, the user can check a summary of the call content.

[1156] User's embodiment

[1157] When a user receives a notification from the server during a call, they open the application and check a summary of the call. For example, they might see something like, "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely fraudulent, they can quickly end the call and report it to the police or emergency contacts. A visual alert displayed on the smart device also allows for quick action.

[1158] Hardware and Software

[1159] Hardware:

[1160] server

[1161] telephone terminal

[1162] Smart devices (e.g., smart glasses, smartphones)

[1163] software:

[1164] Speech recognition: speech_recognition

[1165] Translation: googletrans

[1166] Text analysis: textblob

[1167] Specific examples

[1168] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for surgery costs. When this call begins, the device automatically transfers the call contents to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If the server determines that the call is likely to be fraudulent, it generates an alert message and sends it to Person A and his / her emergency contacts. Person A receives the notification during the call, checks the content, and determines that it is fraudulent. By ending the call and reporting it to the police, the fraud can be prevented. In addition, a visual alert is displayed on smart devices such as smart glasses, so the warning can be received immediately even if the person is away from the phone.

[1169] Prompt Sentence Examples

[1170] If a user wants to add fraud keywords to a generative AI model, here is an example prompt:

[1171] "Add new scam keyword: {new keyword}"

[1172] for example:

[1173] "Add new fraud keywords: bank account, PIN, payment due date"

[1174] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1175] Step 1:

[1176] The terminal detects that there is an incoming call to the telephone terminal. If the call is from a number not registered in the phone book, the terminal transfers the call to the server, which then sends the voice data as input data to be provided to the server.

[1177] Step 2:

[1178] The server converts the received voice data into text data in real time using a speech recognition tool. This conversion process uses speech recognition software (speech_recognition). The input is voice data, and the output is text data that expresses the voice in characters.

[1179] Step 3:

[1180] The server analyzes the converted text data using analytical means to determine the possibility of special fraud. This analysis uses a text analysis library (textblob) and a translation library (googletrans). The input is text data, and data calculations are performed to detect fraud keywords and evaluate their likelihood. The output is an evaluation result indicating the possibility of fraud.

[1181] Step 4:

[1182] If the server determines that fraud is likely, it generates an alert message using an alert message generation means. Specifically, it generates a text message summarizing the details of the fraud. The input is the evaluation result and the text data that may be fraudulent, and the output is the alert message.

[1183] Step 5:

[1184] The server sends the generated alert message to the individual and their emergency contacts using a notification mechanism. Additionally, a visual alert is displayed on the smart device (e.g., smart glasses, smartphone). The notification mechanism uses a push notification service. The input is the alert message, and the output is the notification transmission result.

[1185] Step 6:

[1186] The user receives a push notification from their terminal or smart device during a call and checks the content. The user opens the application and reads a summary of the call to determine whether it is fraudulent. The input is the alert message, and the output is the user's decision.

[1187] Step 7:

[1188] If the user determines that there is a high possibility of fraud, they can quickly end the call and report it to the police or an emergency contact. This makes it possible to prevent actual fraud from occurring. The input is the user's judgment, and the output is the reporting action.

[1189] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1190] The present invention relates to a system that analyzes calls received from telephone terminals in real time and issues an alert if a special fraud is suspected. In particular, the present invention improves the accuracy of fraud detection by combining voice data analysis with an emotion engine that recognizes the user's emotions.

[1191] Server-side implementation

[1192] The server receives voice data sent from the terminal in real time. The voice data is converted into text data by a voice recognition means. For example, if someone says over the phone, "Your son has been in a traffic accident and needs urgent surgery," the content is recorded as text: "Your son has been in a traffic accident and needs urgent surgery."

[1193] Next, an analysis means analyzes this text data to determine whether it is likely to be fraud. In this case, keywords such as "traffic accident" and "surgery costs" are detected, and it is determined that there is a high possibility of fraud.

[1194] Additionally, the server is equipped with an emotion engine that analyzes the user's emotional state from the voice data. The emotion engine detects emotions such as stress or anxiety, and provides this to the analysis means as auxiliary information to further strengthen suspicion of fraud.

[1195] The server then uses an alert message generator to create a summary of the potentially fraudulent call. For example, it could summarize the call as "This call is about a person claiming to be your son who was in a car accident and needs money for surgery." An alert message is then sent to the person and their emergency contacts via a notification mechanism. Specifically, the message could read, "Mom's phone has received a potentially fraudulent call. Please check immediately."

[1196] Terminal side embodiment

[1197] When a call comes in from a number not registered in the phone book, the device automatically forwards the call to the server. At this time, it appears to the user as if the call is normal, but since analysis is done on the server, user operation is minimal.

[1198] When a notification from the server arrives at the device, the device notifies the user of the content via push notification. By receiving the notification, the user can check a summary of the call content.

[1199] User's embodiment

[1200] If a user receives a notification from the server during a call, they can open the app and check a summary of the call. For example, they might see something like, "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely to be fraud, they can quickly end the call and report it to the police or an emergency contact.

[1201] Specific examples

[1202] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, the device automatically transfers the contents of the call to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If the server determines that there is a high possibility of fraud, it uses an emotion engine to analyze the user's emotional state, and if suspicious emotions are confirmed, it incorporates this as supplementary information. Finally, an alert message is generated and sent to Person A and their emergency contact. After Person A receives the notification during the call and checks the content, he or she determines that it is a fraud, ends the call, and reports it to the police, thereby preventing fraud from occurring.

[1203] As described above, the present invention provides a specific method and system for detecting special telephone frauds in real time and responding quickly by making full use of an emotion engine.

[1204] The processing flow will be explained below.

[1205] Server-side processing steps

[1206] Step 1:

[1207] The server receives a call start signal from the terminal. When the terminal detects a call from an unregistered number and begins preparations to transfer the call to the server, a call start notification is sent to the server.

[1208] Step 2:

[1209] The server receives the voice data in real time. When a call is started, the device sends the voice data in streaming format to the server, which stores it in a buffer.

[1210] Step 3:

[1211] The server activates the speech recognition means and converts the received voice data into text. For example, the contents of the call may be converted into text such as "Your son was in a traffic accident and needs urgent medical expenses."

[1212] Step 4:

[1213] The server uses an analytical method to analyze the text data, detect fraud-related keywords (e.g., "traffic accident," "surgery costs," etc.), and determine whether there is any suspicion of fraud.

[1214] Step 5:

[1215] The server activates an emotion engine to analyze the user's emotional state (e.g., anxiety, stress, etc.) from the received voice data. This emotional state is treated as auxiliary information to strengthen suspicion of fraud.

[1216] Step 6:

[1217] The server generates an alert message based on the text data and the results of sentiment analysis. For example, a summary such as "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery" is generated.

[1218] Step 7:

[1219] The server will then use the notification method to send an alert message to the individual and their emergency contacts, such as "There was a potentially fraudulent call to your mother's phone. Please check immediately."

[1220] Terminal processing steps

[1221] Step 1:

[1222] The device receives a call from a number that is not registered in the phone book. When a call comes in from an unregistered number, the call is automatically forwarded to the server. When the call starts, the voice data is sent to the server.

[1223] Step 2:

[1224] The device sends call data to the server in real time. The call audio is recorded on the device and transferred to the server in streaming format.

[1225] Step 3:

[1226] The device waits for and receives alert notifications from the server. If the server determines that fraud is suspected and generates an alert, the notification is sent to the device.

[1227] Step 4:

[1228] The device will display an alert message to the user. When a push notification is received, the notification will say, "Warning: There has been a potentially fraudulent call. Please check the details."

[1229] User processing steps

[1230] Step 1:

[1231] The user receives a notification on the device. When an alert notification arrives on the device during a call, it is visible to the user.

[1232] Step 2:

[1233] The user opens the app and checks the summary of the call. The summary displayed in the app reads, "This call is from someone claiming to be your son and he needs money for surgery after a traffic accident."

[1234] Step 3:

[1235] The user should end the call and, if necessary, contact the police or emergency contacts. If there is a strong suspicion of fraud, the user should immediately end the call and take appropriate action, such as reporting the call to the police.

[1236] In this way, a system is realized in which the server, terminal, and user work together to reduce the risk of special fraud. By combining this system with an emotion engine, the accuracy of fraud detection can be improved, enabling more effective fraud countermeasures.

[1237] Example 2

[1238] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1239] Special fraud methods are becoming more sophisticated every year, and it is extremely difficult for victims to recognize fraud in real time, especially in cases of fraud over the phone. As a result, many people fall victim to fraud and suffer huge losses. In addition, it is necessary to accurately analyze the content of fraudulent calls and detect the emotional state of the victim, but conventional systems are inadequate in dealing with this important aspect.

[1240] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1241] In this invention, the server includes means for analyzing voice data received from a telephone terminal, means for converting the voice data into text, means for determining the possibility of special fraud from the text data, means for analyzing the emotional state using the analysis means, means for generating an alert message based on the determination, and means for sending the alert message to the caller and their emergency contacts. This makes it possible to analyze the content of fraudulent calls in real time and quickly issue an alert if there is a high possibility of fraud. Furthermore, by analyzing the victim's emotional state, the accuracy of fraud detection can be improved, helping to prevent damage before it occurs.

[1242] "Server" refers to a computer system that analyzes and manages voice data received from telephone terminals, and determines and notifies users of the possibility of fraud.

[1243] "Voice data" refers to the voice signal received from the telephone terminal, which is the subject of analysis.

[1244] "Speech recognition means" refers to the process or technology that converts voice data into text data, and a speech recognition engine is an example of this.

[1245] "Text data" refers to data in sentence format converted by a speech recognition means, and serves as the basis for analysis.

[1246] "Analysis methods" refers to the processes and technologies that analyze text data, detect specific keywords and phrases, and determine the possibility of special fraud.

[1247] "Emotion analysis means" refers to processes or technologies that analyze a user's emotional state from voice or text data and provide supplemental information that reinforces the possibility of fraud.

[1248] "Alert Message" refers to a warning message summarizing the content of a call that is determined to be highly likely to be fraudulent.

[1249] "Message generation means" refers to the process or technology for creating an alert message based on the results of the analysis means and the sentiment analysis means.

[1250] "Means of Notification" refers to the process or technology by which the generated alert message is sent to the individual and their emergency contacts.

[1251] "Push notification service" refers to a service that sends messages from a server to a user's device in real time.

[1252] "Telephone terminal" refers to a device for making voice calls.

[1253] This invention relates to a system that analyzes calls received from telephone terminals in real time and issues an alert if a special fraud is suspected. Specifically, it improves the accuracy of fraud detection by combining voice data analysis with an emotion analysis engine that recognizes user emotions.

[1254] Server-side implementation

[1255] The server receives voice data sent from the telephone terminal in real time. This voice data is converted into text data using a speech recognition means. For example, the voice data is converted into text data using the Google Cloud Speech-to-Text API. As a specific example, if someone says over the phone, "Your son has been in a traffic accident and needs urgent medical expenses," the content of the call is recorded as text as "Your son has been in a traffic accident and needs urgent medical expenses."

[1256] The server then uses a natural language processing engine, such as IBM Watson Natural Language Understanding, to analyze the text data. This process determines whether there is a possibility of fraud. For example, if certain keywords, such as "traffic accident" or "surgery costs," are detected, it is determined that there is a high possibility of fraud.

[1257] The server is also equipped with an emotion analysis engine that analyzes the user's emotional state from the voice data. For example, Emotion AI is used to detect emotions such as stress and anxiety. This emotional data is provided to the analysis means as auxiliary information to further increase the likelihood of fraud.

[1258] Furthermore, the server generates an alert message based on the result of the judgment. The alert message generation means creates a summary such as, "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery." The notification means then sends the alert message to the person and their emergency contact. Specifically, the message sent may read, "There has been a call to your mother's phone that is likely to be fraudulent. Please check immediately."

[1259] Terminal side embodiment

[1260] When a call comes in from a number not registered in the phone book, the device automatically forwards the call to the server. At this time, it appears to the user as if the call is normal, but since analysis is done on the server, user operation is minimal.

[1261] In addition, when a notification from the server arrives on the device, the device will notify the user of the content via push notification. When the notification is received, the user can easily check the content of the notification. For example, a prompt message may be sent saying, "There was a call to your mother's phone that is likely to be fraudulent. Please check it immediately."

[1262] User's embodiment

[1263] When a user receives a notification from the server during a call, they can open a dedicated app and check a summary of the call. For example, a summary such as "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery" may be displayed. If the user determines based on this summary that the call is likely to be fraud, they can immediately end the call and report it to the police or an emergency contact.

[1264] Specific examples

[1265] As a specific example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, Person A's device automatically transfers the voice call to the server. The server receives the voice data and converts it into text using the Google Cloud Speech-to-Text API. It then analyzes the text data using IBM Watson Natural Language Understanding to detect keywords such as "traffic accident" and "surgery costs." If the server determines that there is a high possibility of fraud, it uses Emotion AI to analyze the user's emotional state, and if emotions such as stress or anxiety are detected, it incorporates this as supplemental information.

[1266] Finally, the server generates an alert message and sends it to Person A and his / her emergency contacts. After Person A receives the notification during the call and checks the summary, he / she determines that the call is likely a scam, immediately ends the call, and reports it to the police. This prevents the scam from occurring.

[1267] As described above, the present invention provides a specific method and system for detecting special telephone frauds in real time and responding quickly by making full use of an emotion analysis engine.

[1268] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1269] Step 1:

[1270] When a terminal receives a call from a number not registered in the phone book, it automatically transfers the call contents to the server. It receives voice data from the telephone terminal as input and sends this voice data to the server as output. From the user's side, it appears as if a normal call is being made, but in reality, the call contents are being sent to the server in real time.

[1271] Step 2:

[1272] The server receives voice data sent from the terminal in real time. It receives voice data from the terminal as input and passes it to the voice recognition means as output. Specifically, the voice data is sent to the server in streaming format, and the server supplies it to the voice recognition engine.

[1273] Step 3:

[1274] The server converts the received voice data into text data using a speech recognition method (for example, Google Cloud Speech-to-Text API). As input, it receives real-time voice data and converts it into text data. The output is the converted text data. For example, a message such as "Your son has been in a traffic accident and urgently needs money for surgery" may be recorded as text.

[1275] Step 4:

[1276] The server analyzes the text data using analytical means (for example, IBM Watson Natural Language Understanding). It receives the text data as input and detects specific keywords or phrases contained within it. The output is a determination result as to whether or not the keyword or phrase is included. For example, if keywords such as "traffic accident" or "surgery costs" are detected, it determines that the call is likely to be fraudulent.

[1277] Step 5:

[1278] The server uses emotion analysis means (e.g., Emotion AI) to analyze the user's emotional state from the voice data. As input, it receives the original voice data and text data and detects emotions such as stress and anxiety. The output is the detected emotion data. For example, if high anxiety or stress is detected, this is used as auxiliary information to strengthen the possibility of fraud.

[1279] Step 6:

[1280] The server generates an alert message based on the analysis results and sentiment analysis results. It receives the text analysis results and sentiment analysis data as input, and creates a summarized warning message using the alert message generation means. For example, it generates a summary such as "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery." The output is this summary message.

[1281] Step 7:

[1282] The server sends the generated alert message to the user and emergency contacts using a notification method. It receives the generated alert message as input and sends it as output to the user and emergency contacts via email, SMS, or push notification services. Specifically, it sends a message saying, "There was a likely fraudulent call on your mother's phone. Please check it immediately."

[1283] Step 8:

[1284] If a user receives an alert notification during a call, the user opens a dedicated app and checks a summary of the call. The input is receiving the alert notification and displaying the notification content within the application. The output is summary information that is displayed on the user's screen. For example, it might say, "This call is about a person claiming to be your son who was in a traffic accident and needs money for surgery."

[1285] Step 9:

[1286] If the user determines based on this summary information that the call is likely to be fraudulent, they can immediately end the call and report it to the police or emergency contacts. The input is to check the alert notification and summary information and take appropriate action. The output is to end the call and report it to the police. For example, a fraudulent call can be reported using the "Report" button provided in the dedicated app.

[1287] (Application example 2)

[1288] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1289] The methods of special frauds using telephone terminals are becoming more sophisticated, and existing systems may have difficulty detecting them. Furthermore, systems that ignore the user's emotional state have the problem of being unable to adequately determine the possibility of fraud. Therefore, there is a need for a system that can detect fraud with greater accuracy by analyzing both voice data and the user's emotional state.

[1290] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1291] In this invention, the server includes means for analyzing voice data received from a telephone terminal, means for converting the voice data into text, means for determining the possibility of special fraud from the text data, means for generating an alert message, means for sending the alert message to the user and an emergency contact, emotion analysis means for detecting the user's emotional state from the voice data, and means for using the output of the emotion analysis means as supplementary information for the determination. This makes it possible to analyze both the voice data and the user's emotional state and detect the possibility of fraud with higher accuracy.

[1292] A "server" is a central computer system that analyzes voice data received from telephone terminals and determines the possibility of fraud.

[1293] "Speech recognition means" refers to a technique or device for converting received voice data into text data.

[1294] "Analysis means" refers to a technology or device for analyzing text data to determine the possibility of special fraud.

[1295] The "alert message generating means" is a technology or device that, when the analysis means determines that there is a possibility of fraud, generates a message to notify the user or emergency contact based on that information.

[1296] "Notification means" refers to the technology or device used to send the generated alert message to the individual and their emergency contacts.

[1297] An "emotion analysis means" is a technique or device for detecting a user's emotional state (for example, stress or anxiety) from voice data.

[1298] "Supplementary information" is information for incorporating data on the user's emotional state obtained by the emotion analysis means into the determination process of the analysis means.

[1299] The present invention provides a detailed description of a system that detects special frauds in real time via telephone terminals and sends alerts to users and emergency contacts.

[1300] Server embodiment

[1301] The server receives voice data sent from the telephone terminal in real time. The hardware required is a server machine with a high-performance processor and sufficient memory, and the software uses the "Google Speech Recognition API" as the voice recognition engine, the "EmotionalEngine" as the emotion analysis engine, and the "AlertSystem" as the alert system. The server's processing procedure is as follows:

[1302] First, the server receives the voice data and converts it into text data using a voice recognition device. For example, if the caller says, "My son was in a traffic accident and needs urgent surgery money," the content of the call will be recorded as text data.

[1303] Next, an analysis means analyzes this text data to determine the possibility of fraud. At this stage, specific keywords such as "traffic accident" or "surgery costs" are detected. The server also has an emotion analysis means that detects the user's emotional state (e.g., stress or anxiety) from the voice data. This information is provided to the analysis means as supplementary information to further strengthen suspicion of fraud.

[1304] Based on the analysis results, the alert message generating means creates a summary of the potentially fraudulent call, and the notification means sends an alert message to the user and emergency contacts. Specifically, a message such as "There has been a potentially fraudulent call to your mother's phone. Please check it immediately" is sent.

[1305] Terminal embodiment

[1306] The device has a function that automatically forwards calls from numbers not registered in the phone book to the server. At this time, it appears to the user as if they are making a normal call, but the content of the call is analyzed by the server, so the user's operation is minimal.

[1307] When a notification from the server arrives at the device, the device notifies the user of the content via push notification. By receiving the notification, the user can check a summary of the call content.

[1308] User's embodiment

[1309] If a user receives a notification from the server during a call, they can open the "Fraud Guard" application installed on their smartphone and check a summary of the call. For example, it might say, "This call is about someone claiming to be your son who was in a traffic accident and needs money for surgery." If the user determines based on this summary that the call is likely to be fraud, they can quickly end the call and report it to the police or an emergency contact.

[1310] Specific examples

[1311] As a concrete example, consider the case where Person A receives a phone call saying, "My son was in a traffic accident," and is asked for the cost of surgery. When this call begins, the device automatically transfers the contents of the call to the server. The server receives the voice data and converts it into text. It then analyzes the text data and detects keywords such as "traffic accident" and "surgery costs." If it determines that there is a high possibility of fraud, the server performs an emotion analysis, and if it confirms that the user is in a state of high stress, it incorporates this as supplementary information. Finally, an alert message is generated and sent to Person A and their emergency contacts.

[1312] Prompt Sentence Examples

[1313] Example prompts to input to a generative AI model:

[1314] Perform voice analysis, and if the converted text contains words like "traffic accident" or "surgery costs," use emotion analysis to check for stress or anxiety, and generate a program that sends an alert to the user if there is a suspicion of fraud.

[1315] As described above, the present invention provides a specific method and system for detecting special telephone frauds in real time and responding quickly by making full use of an emotion engine.

[1316] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1317] Step 1:

[1318] The telephone terminal receives the incoming call.

[1319] Input: Audio data when a phone call is received.

[1320] Specific operation: When a call is received from an unregistered number, the device starts the call and transfers the voice data to the server in real time.

[1321] Output: Audio data transferred to the server.

[1322] Step 2:

[1323] The server receives the audio data.

[1324] Input: Audio data transferred from the device.

[1325] Specific operation: The server receives audio data in real time using a high-performance data stream processing device.

[1326] Output: The audio data is saved to the server's storage device.

[1327] Step 3:

[1328] The server converts the voice data into text data.

[1329] Input: Stored audio data.

[1330] What it does: Uses the Google Speech Recognition API to convert audio data into corresponding text data.

[1331] Output: Text data.

[1332] Step 4:

[1333] The server analyzes the text data and determines whether it is a special fraud.

[1334] Input: The converted text data.

[1335] Specific operation: Searches for specific keywords (e.g., "traffic accident" or "surgery costs") in the text data. If the conditions are met, it is determined to be a possible fraud.

[1336] Output: Determination result regarding likelihood of fraud.

[1337] Step 5:

[1338] The server detects the user's emotional state from the voice data.

[1339] Input: Received audio data.

[1340] Specific behavior: Uses EmotionalEngine to detect the user's emotional state (e.g., stress or anxiety) from audio data.

[1341] Output: Detected user emotional state data.

[1342] Step 6:

[1343] The server uses the emotional state data as supplementary information for analysis.

[1344] Input: Emotional state data and analysis results.

[1345] What it does: If the user is in a state of high stress or anxiety, that information is used to supplement the analysis results and make a stronger judgment about the likelihood of fraud.

[1346] Output: The final fraud verdict taking into account the supplementary information.

[1347] Step 7:

[1348] The server generates an alert message.

[1349] Input: Final fraud determination result.

[1350] Specific action: Based on the judgment result, a specific alert message (e.g., "This call may be fraudulent. Please check the content.") is automatically generated.

[1351] Output: The alert message.

[1352] Step 8:

[1353] The server sends an alert message via a notification means.

[1354] Input: The generated alert message.

[1355] Specific operation: Uses push notification service to send an alert message to the user and their emergency contacts.

[1356] Output: Sending an alert message to the user and emergency contacts.

[1357] Step 9:

[1358] The user acknowledges the alert message.

[1359] Input: The received alert message.

[1360] Specific actions: Open the Fraud Guard app installed on your smartphone and check the call summary and alert message.

[1361] Output: The user becomes aware of the possibility of fraud and takes necessary measures.

[1362] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1363] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1364] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1365] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1366] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1367] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1368] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1369] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1370] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1371] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1372] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1373] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1374] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1375] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1376] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1377] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1378] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1379] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1380] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1381] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1382] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1383] The following is further disclosed regarding the above embodiment.

[1384] (Claim 1)

[1385] a server that analyzes voice data received from a telephone terminal;

[1386] a speech recognition means for converting the speech data into text;

[1387] An analysis means for determining the possibility of special fraud from the text data;

[1388] means for generating an alert message based on the determination;

[1389] a notification means for sending the alert message to the person and an emergency contact;

[1390] A system including:

[1391] (Claim 2)

[1392] 2. The system of claim 1, wherein said server includes means for automatically forwarding calls from numbers not listed in the phone book.

[1393] (Claim 3)

[1394] The system of claim 1 , wherein the notification means includes means for sending an alert message using a push notification service.

[1395] (Claim 4)

[1396] 2. The system according to claim 1, wherein the analyzing means includes means for detecting keywords from the contents of the call and determining whether fraud is suspected.

[1397] (Claim 5)

[1398] 2. The system of claim 1, wherein the telephone terminal includes means for holding the call content only if the call is determined to be an alert.

[1399] "Example 1"

[1400] (Claim 1)

[1401] a server that analyzes voice data received from a telephone terminal in real time;

[1402] a speech recognition means for converting the speech data into text;

[1403] An analysis means for determining the possibility of special fraud from the text data;

[1404] means for generating a summary of the call based on said determination;

[1405] notification means for sending a summary of said call to the person and an emergency contact;

[1406] A system including:

[1407] (Claim 2)

[1408] 2. The system of claim 1, wherein the server includes means for automatically recording and forwarding calls from numbers not registered in the phone book.

[1409] (Claim 3)

[1410] 10. The system of claim 1, wherein the notification means includes means for sending a summary of the call using a push notification service.

[1411] "Application Example 1"

[1412] (Claim 1)

[1413] a server that analyzes voice data received from a telephone terminal;

[1414] a speech recognition means for converting the speech data into text;

[1415] An analysis means for determining the possibility of special fraud from the text data;

[1416] means for generating an alert message based on the determination;

[1417] a notification means for sending the alert message to the person and an emergency contact;

[1418] a means of displaying visual alerts on smart devices;

[1419] A system including:

[1420] (Claim 2)

[1421] 2. The system of claim 1, wherein said server includes means for automatically forwarding calls from numbers not listed in the phone book.

[1422] (Claim 3)

[1423] The system of claim 1 , wherein the notification means includes means for sending an alert message using a push notification service.

[1424] "Example 2: Combining Emotion Engines"

[1425] (Claim 1)

[1426] a server that analyzes voice data received from a telephone terminal;

[1427] a speech recognition means for converting the speech data into text;

[1428] An analysis means for determining the possibility of special fraud from the text data;

[1429] emotion analysis means for analyzing an emotional state by the analysis means;

[1430] a message generating means for generating an alert message based on the determination;

[1431] a notification means for sending the alert message to the person and an emergency contact;

[1432] A system including:

[1433] (Claim 2)

[1434] 2. The system of claim 1, wherein said server includes means for automatically forwarding calls from numbers not listed in the phone book.

[1435] (Claim 3)

[1436] The system of claim 1 , wherein the notification means includes means for sending an alert message using a push notification service.

[1437] "Application example 2 when combining emotion engines"

[1438] (Claim 1)

[1439] a server that analyzes voice data received from a telephone terminal;

[1440] a speech recognition means for converting the speech data into text;

[1441] An analysis means for determining the possibility of special fraud from the text data;

[1442] means for generating an alert message based on the determination;

[1443] a notification means for sending the alert message to the person and an emergency contact;

[1444] emotion analysis means for detecting an emotional state of a user from voice data;

[1445] means for utilizing an output of the emotion analysis means as supplementary information for the determination;

[1446] A system including:

[1447] (Claim 2)

[1448] 2. The system of claim 1, wherein said server includes means for automatically forwarding calls from numbers not listed in the phone book.

[1449] (Claim 3)

[1450] The system of claim 1 , wherein the notification means includes means for sending an alert message using a push notification service. [Explanation of symbols]

[1451] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. a server that analyzes voice data received from a telephone terminal; a speech recognition means for converting the speech data into text; An analysis means for determining the possibility of special fraud from the text data; means for generating an alert message based on the determination; a notification means for sending the alert message to the person and an emergency contact; A system including:

2. 2. The system of claim 1, wherein said server includes means for automatically forwarding calls from numbers not listed in the telephone directory.

3. The system of claim 1 , wherein the notification means includes means for sending an alert message using a push notification service.

4. 2. The system according to claim 1, wherein said analyzing means includes means for detecting keywords from the contents of the call and determining whether fraud is suspected.

5. 2. The system of claim 1, wherein the telephone terminal includes means for holding the call only if the call is determined to be an alert.

Images