System

Abstract

A system is provided.SOLUTION: A system, comprising: means for collecting data about fraud; means for training an artificial intelligence model to learn patterns of fraud based on the collected data; means for receiving a call from a fraudster and converting the content to text; means for determining a likelihood of fraud using the converted text; means for generating an interactive response if there is a likelihood of fraud; means for converting the generated interactive response to speech and replying to the fraudster; and means for collecting information of the fraudster and sharing the information with an affiliate.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Fraud crimes remain prevalent, and their methods are becoming increasingly sophisticated. In particular, frauds targeting the elderly continue to occur, with many people falling victim to them. Because it is extremely difficult and dangerous for the elderly to deal with fraud themselves, effective prevention measures are needed. Therefore, new technology is needed to prevent fraud crimes, identify fraudsters, and lead to their arrest. [Means for solving the problem]

[0005] The system of the present invention includes a means for collecting data on fraud, a means for training an artificial intelligence model to learn fraud patterns based on the collected data, a means for receiving calls from fraudsters and converting the contents of the calls into text, a means for determining the possibility of fraud using the converted text, a means for generating a dialogue response if there is a possibility of fraud, a means for converting the generated dialogue response into speech and responding to the fraudster, and a means for collecting information on fraudsters and sharing the information with relevant authorities. This system can simulate conversations with fraudsters in real time and efficiently collect and share information for identifying fraudsters. Furthermore, by automatically extracting and collecting characteristic information on fraudsters, it becomes possible to respond to new fraudulent methods.

[0006] "Fraud Data" is information that includes information about the fraudster's methods, means of communication, details of the victim, and the location and time of the fraud.

[0007] "Artificial intelligence model" refers to algorithms and computational techniques that learn specific patterns and anomalies based on collected data and make predictions and classifications.

[0008] A "means for receiving calls" is a system that includes equipment and software that receives and records calls from fraudsters, either locally or remotely.

[0009] "Means for converting to text" refers to technology or devices that analyze received voice data and convert it into text data.

[0010] "Means for determining likelihood of fraud" refers to technology that includes criteria and algorithms for analyzing the converted text and determining whether it is fraudulent.

[0011] "Means for generating a dialogue response" refers to an algorithm or system that automatically generates a response to a fraudster when a possible fraud is determined.

[0012] The "means for converting into voice" refers to a system including a device or software for converting the generated text data into voice data and uttering it in a natural voice.

[0013] "Means of collecting fraudster information" refers to technologies and systems that collect and store characteristics and identifying information of fraudsters obtained during the course of fraudulent activity.

[0014] "Means for sharing with relevant agencies" refers to technologies, including communication systems and protocols, for sharing collected fraudster information with police and other law enforcement agencies in real time. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11]FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] The system of this invention is designed to prevent and expose fraudulent activities by receiving calls from fraudsters, analyzing the content of the conversation, and reporting it to the relevant authorities. Below, we will explain the specific processing and operation of this system, dividing it into three roles: server, terminal, and user.

[0037] Server Processing

[0038] Data collection:

[0039] The server periodically collects fraud data from fraud databases and other sources on the internet, including information about fraudster tactics, methods of communication, victim details, and the location and time of fraud.

[0040] Training the AI ​​model:

[0041] The server uses the collected data to train artificial intelligence models to identify and learn from fraud patterns, which are then used to update criteria and algorithms for determining potential fraud.

[0042] Fraud judgment:

[0043] The server analyzes the text data sent from the device and uses that data to determine the possibility of fraud using an AI model.

[0044] Response generation:

[0045] If fraud is determined to be likely, the server generates natural responses in an elderly voice that serve to continue the dialogue with the fraudster and elicit further information.

[0046] Data sharing:

[0047] The characteristics and identifying information of fraudsters are collected, collated, and shared with the relevant authorities (police, etc.) in real time, which will contribute to the identification and arrest of fraudsters.

[0048] Terminal handling

[0049] Receiving a call:

[0050] The device will always be available to receive calls using an unused phone number, and will automatically receive and answer calls from scammers.

[0051] Voice Recognition:

[0052] The conversation with the scammer is recorded in real time and converted into text using voice recognition technology, which is then sent to a server.

[0053] Audio conversion and transmission:

[0054] The response text received from the server is converted into an elderly-sounding voice, which generates a natural response that is easy for fraudsters to recognize and sends it to the fraudster.

[0055] User (system administrator) processing

[0056] System Monitoring:

[0057] Users can check whether the system is operating normally through dashboards and monitoring tools that monitor the overall system status, and if an abnormality occurs, they can take prompt action.

[0058] System optimization:

[0059] Based on the collected data and system performance data, users can adjust system parameters as needed, including retraining AI models as new fraud techniques are discovered.

[0060] Specific examples

[0061] To give a concrete scenario, consider a situation where a scammer calls and says, "Your bank account is at risk. We need your immediate attention!"

[0062] Terminal handling:

[0063] The device automatically receives the call and converts the recorded voice into text in real time, which is then sent to the server.

[0064] Server Action:

[0065] The server analyzes the received text, determines it is likely a scam, and then generates an elderly-sounding response to the device: "Sorry, what do I do?"

[0066] Terminal handling:

[0067] The device converts the received response text into speech and responds to the fraudster, then records the fraudster's response again and sends it to a server to collect the fraudster's information.

[0068] In this way, the server, terminal, and user work together to interact with fraudsters, and collect, analyze, and share the information necessary to identify them, thereby realizing a system that prevents fraudulent acts before they occur.

[0069] The processing flow will be explained below.

[0070] Step 1:

[0071] Data collection (server processing)

[0072] The server periodically collects fraud data from internet fraud databases and other sources. This data includes information about fraudsters' tactics, communication methods, victim details, location and time of occurrence. This data is stored in an internal database.

[0073] Step 2:

[0074] AI model training (server processing)

[0075] The server uses the collected data to retrain the artificial intelligence model to learn fraud patterns, which is then used to identify fraud and generate responses the next time.

[0076] Step 3:

[0077] Waiting for and receiving calls (terminal processing)

[0078] The device will always be available to receive calls using an unused phone number. If a scammer calls, the device will automatically answer the call and start the conversation.

[0079] Step 4:

[0080] Speech recognition and text conversion (device processing)

[0081] The device records the fraudster's voice in real time and converts it into text using speech recognition technology, which is then sent to a server.

[0082] Step 5:

[0083] Fraud detection (server processing)

[0084] The server analyzes the text data sent from the device and uses an AI model to determine whether there is a high possibility of fraud. If it is determined that there is a high possibility of fraud, it proceeds to the next step.

[0085] Step 6:

[0086] Response generation (server processing)

[0087] The server generates an appropriate elderly-style response based on the fraudulent scheme, which is used to continue the dialogue with the fraudster. The generated response text is sent to the device.

[0088] Step 7:

[0089] Audio conversion and transmission (terminal processing)

[0090] The device receives the response text from the server and converts it into an elderly-sounding voice using speech synthesis technology. The conversation continues when the device responds to the fraudster with the converted voice.

[0091] Step 8:

[0092] Information collection (server processing)

[0093] The server continuously analyzes the information obtained from interactions with the fraudsters and collects their characteristics and specific information (phone number, methods, etc.) in a database.

[0094] Step 9:

[0095] Information sharing (server processing)

[0096] The server collects and analyzes information on fraudsters and shares it with relevant agencies such as the police in real time, which contributes to the identification and arrest of fraudsters.

[0097] Step 10:

[0098] System monitoring (user actions)

[0099] Users regularly check the dashboard to monitor the system's operating status, and if an abnormality or error occurs, they can quickly take action.

[0100] Step 11:

[0101] System optimization (user processing)

[0102] Users analyze system performance data and adjust configuration parameters as needed, as well as retrain AI models to adapt to new fraudulent schemes.

[0103] Example 1

[0104] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0105] Fraudulent phone calls have become a social problem, with the methods becoming more sophisticated and the damage caused by them increasing. Fraud targeting the elderly in particular often leads to serious consequences. Furthermore, fraudsters often use international networks, making it difficult to deal with them using conventional methods. For this reason, there is a need for early detection and response to fraudulent phone calls, but current systems have difficulty determining fraud in real time, gathering information through dialogue, and identifying fraudsters. This article presents a system to solve this problem.

[0106] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0107] In this invention, the server includes means for collecting data on fraud, means for training an artificial intelligence model that learns fraud patterns based on the collected data, means for receiving calls from fraudsters and converting the content of the calls into text data, means for determining the possibility of fraud using the converted text data, means for generating a dialogue response using natural language generation technology if there is a possibility of fraud, means for converting the generated dialogue response into voice and replying to the fraudster, and means for organizing the characteristics and information of the fraudster and sharing that information with the relevant authorities in real time. This enables immediate identification of fraudulent calls, information collection through real-time dialogue, and rapid information sharing with the relevant authorities.

[0108] "Fraud Data" refers to information relating to fraudulent activity, such as the modus operandi, means of communication, victim details, location and time of occurrence.

[0109] "Artificial intelligence model" refers to a program that uses machine learning algorithms to learn patterns of fraudulent activity and determine the likelihood of fraud.

[0110] "Character data" refers to data converted from audio into text format.

[0111] "Natural language generation technology" refers to technology that uses AI to generate natural human conversational sentences.

[0112] "Dialogue Response" refers to text or audio generated to respond in a conversation with the fraudster.

[0113] "Relevant Authorities" refers to the police and other legal authorities that monitor and investigate fraudulent activities.

[0114] "Real-time sharing" refers to immediately transmitting information about fraudulent activity to relevant authorities.

[0115] MODE FOR CARRYING OUT THE INVENTION

[0116] The system of this invention is a system that receives fraudulent phone calls, analyzes the content of the calls, and reports them to the relevant authorities to prevent fraud. Below, we will explain in detail the roles and specific processes of the server, terminal, and user.

[0117] Server Processing

[0118] Data collection:

[0119] The server periodically collects data from fraud databases and other sources on the internet, including information on the modus operandi, means of communication, victim details, location and time of occurrence, using information providers such as "Fraud Risk Database" and "ScanWatch."

[0120] Training the AI ​​model:

[0121] The server uses the collected data to train artificial intelligence models using machine learning frameworks such as TensorFlow and PyTorch, which are then used to build algorithms that learn fraud patterns and determine likelihood of fraud.

[0122] Fraud judgment:

[0123] The server receives the text data of the incoming call sent from the device. This data is analyzed using natural language processing tools such as NLTK and spacy. The AI ​​model determines the likelihood of fraud and, if there are signs of fraud, proceeds with further processing.

[0124] Response generation:

[0125] If fraud is deemed likely, the server uses a generative AI model such as GPT-4 to generate a natural-sounding elderly-sounding response that continues the dialogue with the fraudster and elicits further information.

[0126] Data sharing:

[0127] The server collects the characteristics and identifying information of fraudsters and shares it with the relevant authorities (police, etc.) in real time. This process uses a "real-time data sharing API" and a "RESTful API" to enable rapid data transmission.

[0128] Terminal handling

[0129] Receiving a call:

[0130] The device uses VoIP technology to keep unused phone numbers constantly available for calls, and automatically receives and records calls from scammers.

[0131] Voice Recognition:

[0132] The recorded voice is converted into text data in real time using Google Cloud Speech-to-Text technology, which is then sent to a server.

[0133] Audio conversion and transmission:

[0134] The response text received from the server is converted into an elderly-sounding voice using a speech synthesis service such as Amazon Polly, which is then sent back to the scammer to continue the conversation.

[0135] User (system administrator) processing

[0136] System Monitoring:

[0137] Users monitor the overall operation of the system through dashboards and monitoring tools such as Grafana and Nagios, and if an abnormality occurs in the system, users can take prompt action.

[0138] System optimization:

[0139] Users can adjust system parameters based on collected data and system performance data, and retrain the AI ​​model using Jupyter Notebook or Google Colab as new fraud schemes are discovered.

[0140] Specific examples

[0141] To give a concrete scenario, consider a situation where a scammer calls and says, "Your bank account is at risk. We need your immediate attention!"

[0142] 1. Your device will automatically receive this call and record the audio.

[0143] 2. The recorded audio is converted into text in real time using Google Cloud Speech-to-Text and sent to the server.

[0144] 3. The server analyzes the received text and uses an AI model to determine whether it is likely to be fraudulent.

[0145] 4. Next, generate an elderly-sounding response, "Sorry, what can I do?", and send it to the device.

[0146] 5. The device converts the received text into speech using Amazon Polly and responds to the scammer.

[0147] 6. The scammer's responses are recorded again and sent to the server to collect the scammer's information.

[0148] This process allows for the collection of necessary information while maintaining a dialogue with the scammer, enabling prompt reporting to the appropriate authorities. An example prompt is, "A scammer has called an elderly person to inform them of a risk to their bank account. Please generate a natural response that sounds like an elderly person."

[0149] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0150] Step 1:

[0151] Input: The server collects fraud data from fraud databases and other sources on the Internet.

[0152] How it works: The server periodically retrieves data using designated APIs and web scraping techniques, specifically using a "fraud risk database" and "information collection tools."

[0153] Output: The collected data is stored in a database on the server.

[0154] Step 2:

[0155] Input: Collected fraud data

[0156] How it works: The server preprocesses the collected data, which includes cleaning it, imputing missing values, and normalizing it. It then trains an artificial intelligence model using machine learning frameworks like TensorFlow and PyTorch.

[0157] Output: A trained AI model is generated.

[0158] Step 3:

[0159] Input: Call from scammer

[0160] How it works: The device uses VoIP technology to receive calls from scammers. If a call comes in from a scammer, the device automatically receives and records the call.

[0161] Output: Recorded audio data is obtained.

[0162] Step 4:

[0163] Input: Recorded audio data

[0164] How it works: The device uses Google Cloud Speech-to-Text to convert recorded audio data into text data in real time.

[0165] Output: The converted character data is generated and sent to the server.

[0166] Step 5:

[0167] Input: Character data sent from the terminal

[0168] How it works: The server uses NLTK and spacy to process the received text data and uses AI models to determine the likelihood of fraud. For example, a sentence like "Your bank account is at risk" would be considered a sign of fraud.

[0169] Output: The result is that there is a high possibility of fraud.

[0170] Step 6:

[0171] Input: Character data when it is determined that there is a high possibility of fraud

[0172] How it works: The server uses a generative AI model such as GPT-4 to generate natural-sounding elderly-sounding responses, such as "I'm sorry, what should I do?"

[0173] Output: The generated dialogue response text is generated and sent to the terminal.

[0174] Step 7:

[0175] Input: Dialogue response text sent from the server

[0176] How it works: The device uses a speech synthesis service such as Amazon Polly to convert dialogue response text into an elderly-sounding voice.

[0177] Output: The generated audio data is obtained and sent to the impostor.

[0178] Step 8:

[0179] Input: Scammer's voice response data

[0180] How it works: The scammer's responses are recorded again and converted to text using Google Cloud Speech-to-Text, which is then sent back to the server.

[0181] Output: The scammer's response text data is obtained.

[0182] Step 9:

[0183] Input: Scammer's response text data sent to the server

[0184] How it works: The server compiles the characteristics and identifying information of fraudsters and shares them in real time with relevant authorities (such as the police) using a "real-time data sharing API" or "RESTful API."

[0185] Output: Information about the fraudster that has been shared with the appropriate authorities.

[0186] (Application example 1)

[0187] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0188] Telephone scams by fraudsters are becoming more sophisticated every day, resulting in many victims. Elderly people are particularly vulnerable to fraud, making countermeasures urgently needed. However, current security systems have difficulty detecting fraud in real time and taking countermeasures. There is also the risk that victims may have to confront the fraudsters themselves. Therefore, there is a need to automate the entire process of receiving and identifying fraudulent phone calls, generating dialogue responses, collecting information, and reporting to the relevant authorities, in order to prevent damage before it occurs.

[0189] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0190] In this invention, the server includes: means for collecting data on fraud; means for training an artificial intelligence model to learn fraud patterns based on the collected data; means for receiving calls from fraudsters and converting the content of the calls into text; means for determining the possibility of fraud using the converted text; means for generating a dialogue response if there is a possibility of fraud; means for converting the generated dialogue response into voice and responding to the fraudster; means for collecting information on the fraudster and sharing the information with relevant institutions; means for the smart device to receive calls and convert the voice into text in real time; and means for issuing a warning to the user based on the determination result and generating an elderly-style response. This automates the process from receiving fraudulent calls to determining whether they are fraudulent, generating a dialogue response, collecting information, and reporting to relevant institutions, making it possible to protect victims, especially elderly people, from fraud.

[0191] "Fraud Data" means information relating to fraudulent activity, including data about the fraudster's methods, means of communication, details of the victim, and the location and time of the fraud.

[0192] An "artificial intelligence model" is an algorithm that learns patterns from large amounts of data and determines the likelihood of fraud based on new data.

[0193] "Means for converting to text" refers to the process of converting voice data into text data using voice recognition technology.

[0194] The "means for determining the likelihood of fraud" is an algorithm or machine learning model that analyzes the collected text data and determines whether it constitutes fraudulent activity.

[0195] The "means for generating a dialogue response" is a process that automatically creates an appropriate response to continue dialogue with the fraudster when it is determined that there is a high possibility of fraud.

[0196] The "means for converting to speech" is a speech synthesis technology for converting the generated text dialogue response into speech output.

[0197] "Means for sharing with relevant authorities" refers to a data transmission function that allows information about fraudsters to be reported to relevant authorities and security agencies.

[0198] "Means by which a smart device receives a call" refers to the process by which a smartphone or other mobile device receives a call from a scammer and initiates processing.

[0199] "Means for converting to text in real time" refers to speech recognition technology that instantly converts received speech into text data.

[0200] The "means of issuing a warning to the user and generating an elderly-sounding response" is a mechanism that, if it is determined that there is a high possibility of fraud, displays a fraud warning to the user and generates a response that allows the user to continue a natural dialogue with the fraudster.

[0201] The system of this invention provides a specific process for automatically detecting fraudulent calls and taking countermeasures. This system is mainly composed of three parts: a server, a terminal, and a user.

[0202] Server Processing

[0203] 1. Data Collection:

[0204] The server periodically collects fraud data from fraud databases and other sources on the internet, including information about fraudsters' tactics, methods of communication, victim details, and location and time of occurrence.

[0205] 2. Training the AI ​​model:

[0206] The server trains artificial intelligence models to identify and learn from the collected data, which are then used to update criteria and algorithms for determining potential fraud.

[0207] 3. Fraud determination:

[0208] The server analyzes the text data sent from the device during the incoming call, and uses that data to determine the possibility of fraud using an AI model.

[0209] 4. Response Generation:

[0210] If fraud is determined to be likely, the server generates natural responses in an elderly voice that serve to continue the dialogue with the fraudster and elicit further information.

[0211] 5. Data Sharing:

[0212] The characteristics and identifying information of fraudsters are collected, collated, and shared with the relevant authorities (police, etc.) in real time, which will contribute to the identification and arrest of fraudsters.

[0213] Terminal handling

[0214] 1. Receiving a call:

[0215] The device will always be available to receive calls using an unused phone number, and if a call comes in from a scammer, it will automatically receive and answer the call.

[0216] 2. Speech Recognition:

[0217] The conversation with the scammer is recorded in real time and converted into text using speech recognition technology. This converted text data is sent to a server, which allows the received voice to be instantly converted into text. The hardware used could be a smartphone or mobile device, and the software could be Google Speech-to-Text API or IBM Watson Speech to Text.

[0218] 3. Audio conversion and transmission:

[0219] The response text received from the server is converted into an elderly-sounding voice, which generates a natural response that is easy for fraudsters to recognize and sends it to them. The voice synthesis technology used is gTTS (Google Text-to-Speech).

[0220] User (system administrator) processing

[0221] 1. System Monitoring:

[0222] Users can check whether the system is operating normally through dashboards and monitoring tools that monitor the overall system status, and if an abnormality occurs, they can take prompt action.

[0223] 2. System optimization:

[0224] Users can adjust system parameters based on collected data and system performance data, including retraining AI models as new fraud techniques are discovered.

[0225] Specific examples

[0226] For example, consider the case where a fraudster calls and says, "Your bank account is at risk. We need your immediate attention!" First, the device automatically receives the call and converts the recorded voice into text in real time. This converted text is sent to a server, which analyzes the received text and determines that it is likely a fraud. Next, the device generates an elderly-sounding response such as "I'm sorry, what should I do?" and sends it to the device. The device then converts the received response text into voice and replies to the fraudster. The fraudster's response is recorded again and further information is sent to the server. This series of steps prevents fraud before it occurs.

[0227] Prompt Sentence Examples

[0228] This is an example of actual input to a generative AI model.

[0229] "Please recognize the voice of this call and determine if it is likely a scam. If it is, please display a warning and generate an elderly-sounding response to continue the conversation with the scammer."

[0230] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0231] Step 1:

[0232] Receiving a call

[0233] The terminal always keeps itself available to receive calls using an unused phone number. When a call from a scammer comes in, it automatically answers the call and starts recording. At this point, the input is the audio data of the received call, and the output is the recorded audio data.

[0234] Step 2:

[0235] Voice Recognition

[0236] The device converts the recorded voice data into text in real time using speech recognition technology. The input is the recorded voice data, and the output is text data. This process uses the Google Speech-to-Text API and IBM Watson Speech to Text.

[0237] Step 3:

[0238] Fraud judgment

[0239] The server receives the text data sent from the device and uses an artificial intelligence model to determine the likelihood of fraud. The input is the text data sent from the device, and the output is the determination result regarding the likelihood of fraud. A machine learning algorithm is used for data analysis.

[0240] Step 4:

[0241] Warning notice

[0242] If the server determines that there is a high possibility of fraud, it sends a warning to the terminal. The input is the result of the fraud determination, and the output is a warning to the user. This warning is displayed to the user through the terminal interface.

[0243] Step 5:

[0244] Response Generation

[0245] If the server determines that fraud is likely, it generates a natural response in the style of an elderly person. This generated response serves to continue the dialogue with the fraudster and elicit further information. The input is the result of the fraud detection, and the output is a response sentence in the style of an elderly person. A generative AI model is used to generate this response.

[0246] Step 6:

[0247] Voice conversion and transmission

[0248] The device converts the response text received from the server into an elderly-sounding voice. The voice synthesis technology used is gTTS (Google Text-to-Speech). The input is the generated response text, and the output is voice data. The converted voice is then sent back to the fraudster.

[0249] Step 7:

[0250] Scammers' information gathering and sharing

[0251] The device transmits the recorded voice data during the ongoing conversation with the fraudster to the server, which analyzes it, extracts the fraudster's characteristic information, and shares it with relevant organizations. The input is the recorded voice data, and the output is the analyzed characteristic information of the fraudster and the data to be shared with relevant organizations.

[0252] This series of steps will automate the entire process, from receiving fraudulent calls to identifying them, generating dialogue responses, collecting information, and reporting them to relevant authorities.

[0253] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0254] This invention is a system designed to receive and analyze calls from fraudsters and strengthen fraud prevention measures, and by combining it with an emotion engine to recognize user emotions, it further enhances accuracy. This system consists of three main components: a server, a terminal, and a user. The process and operation of each component are explained in detail below.

[0255] Server Processing

[0256] Data collection

[0257] The server periodically collects fraud data from online fraud databases and other sources. This data includes information about fraudsters' tactics, communication methods, victim details, location and time of occurrence. The collected data is stored in the server's internal database.

[0258] Training an AI model

[0259] Based on the collected data, the server trains an artificial intelligence model to learn fraud patterns, which includes criteria and algorithms to determine the likelihood of fraud.

[0260] Fraud judgment

[0261] The server analyzes the text data of the incoming call sent from the device and uses an AI model to determine the possibility of fraud. If it determines that there is a high possibility of fraud, it proceeds to the next step.

[0262] Response Generation

[0263] If fraud is determined to be likely, the server generates a natural-sounding response in the style of an elderly person, which is used to continue the dialogue with the fraudster and extract further information. The generated response text is then sent to the device.

[0264] Data Sharing

[0265] Collect and organize information about the characteristics and identity of fraudsters, and share it with the relevant authorities (police, etc.) in real time. This information sharing will contribute to the identification and arrest of fraudsters.

[0266] Terminal handling

[0267] Receiving a call

[0268] The device will use an unused phone number and will always be available to receive calls. If a scammer calls, the device will automatically receive and answer the call.

[0269] Voice Recognition

[0270] The device records the fraudster's voice in real time and converts it into text using voice recognition technology, which is then sent to a server.

[0271] Voice conversion and transmission

[0272] The response text received from the server is converted into an elderly-sounding voice, and the conversation continues by replying to the fraudster with the converted voice.

[0273] emotion recognition

[0274] The emotion engine installed on the device analyzes the user's emotions during the conversation with the scammer. For example, emotions such as anger, anxiety, and tension can be detected from the scammer's voice. This emotional information is sent to the server and reflected in the response generation.

[0275] User (system administrator) processing

[0276] System Monitoring

[0277] Users can check the dashboard to monitor the overall system status and quickly respond to any abnormalities or errors that occur.

[0278] System Optimization

[0279] Users can adjust system parameters based on collected data and system performance data, and retrain AI models to adapt to new fraudulent schemes.

[0280] Specific examples

[0281] As a specific scenario, let's consider a situation where a scammer calls and says, "Your bank account has been compromised. We need your immediate attention."

[0282] Terminal handling:

[0283] The device automatically receives the call and records the audio, which is then converted into text using voice recognition technology and sent to a server.

[0284] Server Action:

[0285] The server analyzes the received text, determines it is likely a scam, and then generates an elderly-sounding response to the device: "Sorry, what do I do?"

[0286] Terminal handling:

[0287] The device converts the received response text into speech and responds to the fraudster, while simultaneously analyzing the emotions in the fraudster's voice and sending the emotional information to the server.

[0288] Server Action:

[0289] The server uses this emotional information to generate an appropriate response that reflects the fraudster's emotional state when generating the next response. For example, if the fraudster is feeling anxious, the server can use this to devise a response that will further encourage the fraudulent tactics.

[0290] In this way, the server, terminal, and user work together to communicate with fraudsters in order to prevent fraud before it happens. Furthermore, by combining it with an emotion engine, the system can analyze the emotions of fraudsters, enabling more effective dialogue and information gathering.

[0291] The processing flow will be explained below.

[0292] Step 1:

[0293] Data collection (server processing)

[0294] The server periodically collects fraud data from fraud databases on the Internet and other sources. This data includes information about fraudsters' tactics, communication methods, victim details, location and time of occurrence. This collected data is stored in an internal database.

[0295] Step 2:

[0296] AI model training (server processing)

[0297] The server uses the collected data to retrain its artificial intelligence model, which learns fraud patterns, allowing it to adapt to the latest fraud techniques. The trained AI model is then used to identify fraud and generate responses for the next transaction.

[0298] Step 3:

[0299] Waiting for and receiving calls (terminal processing)

[0300] The device will use an unused phone number and will always be available to receive calls. If a scammer calls, the device will automatically answer the call and initiate the call.

[0301] Step 4:

[0302] Speech recognition and text conversion (device processing)

[0303] The device records the fraudster's voice in real time and converts this voice data into text using speech recognition technology, which is then sent to a server.

[0304] Step 5:

[0305] Fraud detection (server processing)

[0306] The server analyzes the text data sent from the device and uses an AI model to determine the likelihood of fraud. If it determines that there is a high possibility of fraud, it proceeds to the next step.

[0307] Step 6:

[0308] Emotion recognition (device processing)

[0309] An emotion engine on the device analyzes the scammer's voice and determines their emotional state (e.g., anger, impatience, calm, etc.) This emotion data is sent to the server for use in generating subsequent responses.

[0310] Step 7:

[0311] Response generation (server processing)

[0312] The server generates appropriate responses in the style of an elderly person based on the fraudster's tactics and emotional data. The responses are intended to continue the dialogue with the fraudster and elicit further information. The generated response text is then sent to the device.

[0313] Step 8:

[0314] Audio conversion and transmission (terminal processing)

[0315] The device uses speech synthesis technology to convert the response text received from the server into an elderly-sounding voice, which is then sent back to the fraudster, continuing the dialogue.

[0316] Step 9:

[0317] Information collection (server processing)

[0318] The server analyzes the information collected through interactions with the fraudsters and stores their characteristics (phone number, tactics, emotional state, etc.) in a comprehensive database.

[0319] Step 10:

[0320] Information sharing (server processing)

[0321] The server then shares the collected and analyzed information on fraudsters with the police and other relevant authorities in real time. This information sharing process facilitates the identification and arrest of fraudsters.

[0322] Step 11:

[0323] System monitoring (user actions)

[0324] Users can use a dashboard to monitor the overall system operation status and respond quickly to any errors or abnormalities that occur. Daily monitoring maintains the reliability and stability of the system.

[0325] Step 12:

[0326] System optimization (user processing)

[0327] Users can analyze the system's performance data and adjust configuration parameters as needed, including instructing the AI ​​model to retrain to adapt to new fraudulent schemes.

[0328] In a specific scenario, if a fraudster calls and says, "Your bank account is at risk. Immediate action is required," the system works as follows: The device receives the call and uses voice recognition technology to convert the content into text and send it to the server. The server analyzes the text, determines whether it is likely to be fraudulent, generates a response text, and sends it to the device. The device then converts the response text into speech and replies to the fraudster. During this process, the emotion engine analyzes the fraudster's emotions, and this data is reflected in the generation of the next response.

[0329] Example 2

[0330] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0331] In recent years, fraud has become more sophisticated and diverse, making it difficult to respond to conventional countermeasures. Telephone fraud targeting the elderly has been increasing in particular, and effective countermeasures are needed. Systems are needed that not only prevent fraud from occurring, but also collect and share information in real time to identify and arrest fraudsters. Furthermore, there is a need to expose fraudulent activities by analyzing fraudsters' emotions and engaging in more effective dialogue.

[0332] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting data related to fraud, means for training an artificial intelligence model to learn fraud patterns based on the collected data, means for receiving communications from fraudsters and converting the content of the communications into text, means for determining the possibility of fraud using the converted text, means for generating a dialogue response if there is a possibility of fraud, means for converting the generated dialogue response into voice and replying to the fraudster, means for analyzing the characteristics of the fraudster and obtaining emotional information, and means for sharing information about the fraudster with relevant agencies. This improves the accuracy of fraud countermeasures, prevents fraudulent acts, and enables the collection and sharing of information to identify and arrest fraudsters in real time.

[0333] "Fraud Data" means information relating to fraudulent activity, including information such as the fraudulent method, means of communication, details of the victim, and the location and time of the occurrence.

[0334] An "artificial intelligence model" refers to an algorithm that learns from data and automatically performs specific tasks.

[0335] "Communications" refers to the means of communication used by scammers, such as voice or messages.

[0336] "Means for converting to text" refers to speech recognition technology for converting voice or messages into text data.

[0337] "Means for determining likelihood of fraud" refers to algorithms or models that analyze the converted text data and assess the risk of fraud.

[0338] "Means for generating dialogue responses" refers to an algorithm or generative model for automatically generating appropriate responses to continue the dialogue with the fraudster.

[0339] "Means for converting to speech" refers to speech synthesis technology for playing the generated text dialogue response as speech.

[0340] "Means of obtaining emotional information" refers to technology that analyzes emotions from the fraudster's voice and messages and identifies emotional states such as anger or impatience.

[0341] "Means of sharing information with relevant agencies" refers to communication methods and interfaces for providing information about fraudsters to relevant agencies, such as the police, in real time.

[0342] This invention is a system designed to prevent fraud by receiving calls from fraudsters, analyzing them, and strengthening countermeasures. The system consists of three main components: a server, a terminal, and a user, which work together.

[0343] Server Processing

[0344] Data collection

[0345] The server collects fraud data from online fraud databases and other sources, including fraudster tactics, communication methods, victim details, location and time of occurrence. Specifically, it retrieves data from the ScamWatch and FraudLog databases using HTTP requests and stores it in the server's internal database.

[0346] Training an AI model

[0347] The collected data is preprocessed and an artificial intelligence model is trained. For example, TensorFlow or PyTorch is used to learn fraud patterns. Specific features such as specific methods and time periods are incorporated into the algorithm to improve the accuracy of fraud detection.

[0348] Analysis of incoming calls and fraud detection

[0349] The server receives the text data of the incoming call sent from the device and analyzes it using an AI model. The analysis uses keywords and phrases contained in the text data to determine the possibility of fraud. For example, phrases such as "bank account," "risk," and "action required" are determined to indicate a high risk of fraud.

[0350] Response Generation

[0351] If fraud is deemed likely, the server uses a generative AI model (e.g., OpenAI GPT-4) to generate natural-sounding responses in the style of an elderly person, designed to continue the dialogue with the fraudster and elicit further information, such as "I'm sorry, what should I do?"

[0352] Data Sharing

[0353] The characteristics and identifying information of fraudsters are collated and shared with the relevant authorities (e.g., police) in real time. This information is sent via API to expedite response.

[0354] Terminal handling

[0355] Receiving a call

[0356] The device is set to an unused phone number and is always ready to receive calls. When a call comes in from a scammer, it automatically receives the call and prepares to record it. Specifically, it uses Twilio's call receiving API.

[0357] Speech recognition and text conversion

[0358] The device records the fraudster's voice in real time and converts it into text data using Google Cloud Speech-to-Text. The recorded audio stream is sent to an API, which outputs the text data, which is then sent to the server.

[0359] Voice conversion and transmission

[0360] The response text received from the server is converted into an elderly-sounding voice. Specifically, speech synthesis is performed using Amazon Polly or Google Text-to-Speech. The converted voice is then sent back to the fraudster, and the dialogue continues.

[0361] emotion recognition

[0362] The device uses the Microsoft Azure Emotion API to analyze emotional information from the fraudster's voice, detecting emotions such as anger, worry, and tension, and sending this emotional information to a server.

[0363] User (system administrator) processing

[0364] System Monitoring

[0365] Users use Grafana and Kibana to monitor the overall system status. If an abnormality or error occurs, they receive notifications and can respond quickly. They can understand the situation by checking log data and real-time notifications.

[0366] System Optimization

[0367] Users can adjust system parameters based on collected data and system performance data, and retrain the AI ​​model as needed to update the system to adapt to new fraudulent schemes.

[0368] Specific examples

[0369] As a specific scenario, let's consider a situation where a scammer calls and says, "Your bank account has been compromised. We need your immediate attention."

[0370] Terminal handling

[0371] The device automatically receives the call using Twilio, then converts the fraudster's speech into text in real time using Google Cloud Speech-to-Text, which then sends the text data to the server via an RTP stream.

[0372] Server Processing

[0373] The server receives the text data, "Your bank account is at risk. Immediate action is required." and analyzes it using an AI model. If it determines there is a high possibility of fraud, it uses OpenAI's GPT-4 to generate an elderly-sounding response, "I'm sorry, what should I do?", and sends it to the device.

[0374] Terminal handling

[0375] The device uses Amazon Polly to convert the received response text into an elderly-sounding voice and responds to the fraudster. At the same time, it uses Microsoft Azure's Emotion API to analyze emotional information (such as impatience or tension) from the fraudster's voice and sends it to the server.

[0376] Server Processing

[0377] The server will use this emotional information to generate the next response. For example, if it determines that the fraudster is anxious, it can generate a question like, "I'd like to confirm again, is your account really at risk?" This can further refine the fraudulent scheme.

[0378] Example prompts to input to the generative AI model

[0379] "Please translate the following sentence into a natural response from a senior citizen: 'Your bank account is at risk. Action is required immediately.'"

[0380] This prompt can be used to automatically generate an appropriate response.

[0381] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0382] Server Processing

[0383] Step 1: Data collection

[0384] The server collects fraud data from fraud databases and other sources on the internet. Specifically, it uses HTTP requests to retrieve information from the database about the fraud scheme, communication method, victim details, location and time of the incident. The input is the database API endpoint, and the output is detailed data about the fraud. The retrieved data is stored in the server's internal database.

[0385] Step 2: Training the AI ​​model

[0386] The server preprocesses the collected data and trains an artificial intelligence model. Specifically, it uses TensorFlow and PyTorch to learn fraud patterns. The input is the preprocessed dataset, and the output is a trained AI model. Features such as specific methods and time periods are reflected in the algorithm to improve the model's accuracy.

[0387] Step 3: Analyze incoming calls and identify fraud

[0388] The server receives the text data of the incoming call sent from the device and analyzes it using an AI model. The input is text data, and the output is a judgment result indicating the possibility of fraud. In this process, keywords and phrases contained in the text are analyzed to assess the risk of fraud.

[0389] Step 4: Response Generation

[0390] If the server determines that there is a high possibility of fraud, it uses a generative AI model to generate a natural-sounding response in the style of an elderly person. The input is the judgment result and the fraudster's text data, and the output is the generated response text. For example, it generates a response such as, "I'm sorry, what should I do?"

[0391] Step 5: Data sharing

[0392] The server compiles the characteristics and identifying information of fraudsters and shares it with the relevant authorities in real time. The input is detailed information about the fraudsters, and the output is the information sent to the relevant authorities. The information is sent quickly through the API, contributing to the identification and arrest of fraudsters.

[0393] Terminal handling

[0394] Step 1: Receiving a call

[0395] The terminal is set to an unused phone number and is always ready to receive calls. The input is a call from a fraudster, and the output is the received voice data. Specifically, it uses Twilio's call receiving API to automatically receive calls.

[0396] Step 2: Speech recognition and text conversion

[0397] The device records the fraudster's voice in real time and converts it into text data using Google Cloud Speech-to-Text. The input is audio data and the output is text data. The recorded audio stream is sent to the API, and the converted text is sent to the server.

[0398] Step 3: Convert and send

[0399] The device converts the response text received from the server into an elderly-sounding voice. The input is the response text, and the output is synthesized voice data. Specifically, Amazon Polly and Google Text-to-Speech are used to generate the voice and respond to the scammer.

[0400] Step 4: Emotion Recognition

[0401] An emotion engine implemented on the device is used to analyze emotions from the fraudster's voice. The input is the fraudster's voice data, and the output is emotional information (e.g., anger, tension, etc.). The emotional information is sent to the server and reflected in the generation of the next response.

[0402] User (system administrator) processing

[0403] Step 1: System monitoring

[0404] Users check a dashboard to monitor the overall system operation status. The input is real-time system log data, and the output is the monitoring results. Specifically, Grafana and Kibana are used to monitor and receive notifications for abnormalities and errors.

[0405] Step 2: System optimization

[0406] Users adjust the system parameters based on collected data and system performance data. The input is performance data and information on new fraud schemes, and the output is optimized system settings. If necessary, the AI ​​model can also be retrained.

[0407] (Application example 2)

[0408] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0409] The number of fraudulent phone calls, often involving sophisticated tactics by scammers, is steadily increasing, with elderly people being particularly vulnerable to these attacks. Current systems for combating fraudulent phone calls often struggle to identify these calls, resulting in delayed effective responses. Furthermore, they lack the means to analyze the emotions and psychological state of fraudsters, making it difficult to effectively deter and collect information from them. This can result in investigative authorities being unable to respond quickly, potentially exacerbating the damage. Therefore, there is a need for a system that can improve the accuracy and effectiveness of countermeasures against fraudulent phone calls by recognizing the emotions of fraudsters and generating responses that sound like an elderly person.

[0410] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0411] In this invention, the server includes means for collecting data on fraud, means for training an artificial intelligence model that learns fraud patterns based on the collected data, means for determining the possibility of fraud using the converted text, means for generating an elderly-sounding dialogue response if there is a possibility of fraud, means for converting the generated dialogue response into voice and replying to the fraudster, means for analyzing emotions from the fraudster's voice, and means for collecting information on the fraudster and sharing it with relevant institutions. This makes it possible to provide an appropriate response according to the fraudster's emotional state, effectively continue the dialogue to collect information, and ultimately prevent fraud before it occurs.

[0412] "Fraud Data" means any information relating to fraudulent activity, including the method of fraud, means of communication, details of the victim, location and time of the occurrence.

[0413] An "artificial intelligence model" refers to a computer model that can learn from large amounts of data and identify and analyze specific patterns and features.

[0414] "Convert to text" refers to the process of converting audio data into text data.

[0415] "Means for determining the likelihood of fraud" refers to the process of using artificial intelligence models or other algorithms to analyze and determine whether fraud is suspected.

[0416] "Elderly-style dialogue responses" refers to text and voice responses generated to make it appear as if an elderly person is having a conversation.

[0417] "Convert to voice" refers to the process of converting text data into voice data.

[0418] "Means for analyzing emotions from fraudsters' voices" refers to technology for identifying the emotional state (e.g., anger, impatience, tension, etc.) of fraudsters from their vocalizations.

[0419] "Fraudster information" refers to any information that can be used to identify a fraudster, including voice characteristics, phone number, words used, and conversation content.

[0420] "Means of sharing with relevant agencies" refers to the process for providing collected and analyzed fraud information to investigative agencies such as the police and monitoring agencies.

[0421] The present invention is a system for dealing with fraudulent calls from fraudsters, and is composed of a server, a terminal, and a user. The processing of the entire system will be described in detail below.

[0422] Server Processing

[0423] Data collection

[0424] The server periodically collects fraud data from online fraud databases and other sources. This data includes information about the fraud scheme, means of communication, victim details, location and time of occurrence. The collected data is stored in the server's internal database.

[0425] Training an AI model

[0426] Based on the collected data, the server trains an artificial intelligence model to learn fraud patterns, specifically by using pattern recognition techniques and machine learning algorithms to extract and learn the characteristics of suspicious communications.

[0427] Fraud judgment

[0428] The server analyzes the text data of the incoming call sent from the terminal and uses an artificial intelligence model to determine the possibility of fraud. If the result of this determination indicates a high possibility of fraud, the system proceeds to the next step.

[0429] Response Generation

[0430] If the server determines that fraud is likely, it generates a natural response in the style of an elderly person. This response is used to continue the dialogue with the fraudster and extract further information. The generated response text is sent to the device.

[0431] Data Sharing

[0432] Collecting characteristics and identifying information of fraudsters and sharing it with relevant authorities (such as the police) in real time, which will contribute to identifying and arresting fraudsters.

[0433] Terminal handling

[0434] Receiving a call

[0435] The device will always be ready to receive calls, and if a scammer calls you, it will automatically receive and answer the call.

[0436] Voice Recognition

[0437] The device records the fraudster's voice in real time and converts it into text using voice recognition technology (e.g., Google Cloud Speech-to-Text), which is then sent to a server.

[0438] Voice conversion and transmission

[0439] The response text received from the server is converted into an elderly-sounding voice. The converted voice is then sent back to the fraudster, continuing the dialogue. Tools such as Google Text-to-Speech (gTTS) are used for voice synthesis.

[0440] emotion recognition

[0441] An emotion engine (e.g., IBM Watson Tone Analyzer) installed on the device analyzes the emotions of the fraudster during the conversation. The analysis results are sent to the server and reflected in the response generation.

[0442] User (system administrator) processing

[0443] System Monitoring

[0444] Users can check the dashboard to monitor the overall system status, and respond quickly if an abnormality or error occurs.

[0445] System Optimization

[0446] Users can adjust system parameters based on collected data and system performance data, and retrain AI models to adapt to new fraudulent schemes.

[0447] Specific examples

[0448] For example, imagine a scammer calling you and saying, "Your bank account is at risk. We need your immediate attention."

[0449] 1. Terminal Processing

[0450] The device automatically receives the call and records the audio, which is then converted into text using voice recognition technology and sent to a server.

[0451] 2. Server Processing

[0452] The server analyzes the received text and determines that it is likely a scam, then generates an elderly-sounding response to the user, such as "Really? What should I do?", and sends it to the device.

[0453] 3. Terminal Processing

[0454] The device converts the received response text into speech and responds to the fraudster. It also analyzes the emotions in the fraudster's voice and sends the emotional information to the server.

[0455] 4. Server Processing

[0456] The server then uses the emotional information to generate an appropriate response that reflects the fraudster's emotional state. For example, if the fraudster is feeling anxious, the server can use this to devise a response that will further encourage the fraudulent tactics.

[0457] Example prompt sentence:

[0458] "Pretend to be a senior citizen and respond naturally to the following statement: 'Your bank account is at risk.'"

[0459] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0460] Step 1:

[0461] Receiving a call (terminal)

[0462] The device is always ready to receive calls. When a call comes in from a fraudster, it automatically answers the call and starts the conversation. The input is the call signal from the fraudster, and the output is the call audio data.

[0463] Step 2:

[0464] Voice recording and recognition (device)

[0465] The device records the scammer's voice in real time and converts the voice data into text data using voice recognition technology such as Google Cloud Speech-to-Text. The input is the recorded voice data, and the output is text data.

[0466] Step 3:

[0467] Sending text data (terminal → server)

[0468] The terminal sends the converted text data to the server. The input is the text data generated by the speech recognition technology, and the output is the text data sent to the server.

[0469] Step 4:

[0470] Fraud detection (server)

[0471] The server analyzes the received text data and uses an artificial intelligence model (generative AI model) to determine the likelihood of fraud. The input is text data, and the output is a determination result indicating the likelihood of fraud. This data is processed through data calculations and analysis within the server.

[0472] Step 5:

[0473] Generate response (server)

[0474] If the server determines that there is a high possibility of fraud, it generates a natural response in the style of an elderly person for the next dialogue step, where it uses a generative AI model and a prompt to generate an appropriate response. The input is the fraud detection result and the prompt, and the output is the generated response text.

[0475] Step 6:

[0476] Sending a response (server → terminal)

[0477] The server sends the generated response text to the terminal. The input is the generated response text, and the output is the text data sent to the terminal.

[0478] Step 7:

[0479] Voice conversion and reply (terminal)

[0480] The device converts the received response text into an elderly-sounding voice and generates an audio file using Google Text-to-Speech (gTTS), which then replies to the fraudster using this generated voice. The input is the response text, and the output is the voice response to the fraudster.

[0481] Step 8:

[0482] Emotion Recognition (Device)

[0483] The device analyzes the emotions from the fraudster's voice and sends the results to a server, using technologies such as IBM Watson Tone Analyzer. The input is the fraudster's voice data, and the output is the analyzed emotion data.

[0484] Step 9:

[0485] Receiving and responding to emotion data (server)

[0486] The server receives the emotion data sent from the device and reflects it when generating the next response based on that emotion information. This makes it possible to generate an appropriate response based on the fraudster's psychological state. The input is emotion data, and the output is the next response text based on the emotion.

[0487] Step 10:

[0488] Fraudster information collection and sharing (server)

[0489] The server organizes and collects information obtained from fraudsters and shares it with the relevant authorities (police, etc.) in real time. The input is the fraudster information obtained from the conversation, and the output is the shared information sent to the relevant authorities.

[0490] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0491] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0492] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0493] [Second embodiment]

[0494] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0495] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0496] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0497] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0498] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0499] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0500] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0501] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0502] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0503] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0504] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0505] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0506] The system of this invention is designed to prevent and expose fraudulent activities by receiving calls from fraudsters, analyzing the content of the conversation, and reporting it to the relevant authorities. Below, we will explain the specific processing and operation of this system, dividing it into three roles: server, terminal, and user.

[0507] Server Processing

[0508] Data collection:

[0509] The server periodically collects fraud data from fraud databases and other sources on the internet, including information about fraudster tactics, methods of communication, victim details, and the location and time of fraud.

[0510] Training the AI ​​model:

[0511] The server uses the collected data to train artificial intelligence models to identify and learn from fraud patterns, which are then used to update criteria and algorithms for determining potential fraud.

[0512] Fraud judgment:

[0513] The server analyzes the text data sent from the device and uses that data to determine the possibility of fraud using an AI model.

[0514] Response generation:

[0515] If fraud is determined to be likely, the server generates natural responses in an elderly voice that serve to continue the dialogue with the fraudster and elicit further information.

[0516] Data sharing:

[0517] The characteristics and identifying information of fraudsters are collected, collated, and shared with the relevant authorities (police, etc.) in real time, which will contribute to the identification and arrest of fraudsters.

[0518] Terminal handling

[0519] Receiving a call:

[0520] The device will always be available to receive calls using an unused phone number, and will automatically receive and answer calls from scammers.

[0521] Voice Recognition:

[0522] The conversation with the scammer is recorded in real time and converted into text using voice recognition technology, which is then sent to a server.

[0523] Audio conversion and transmission:

[0524] The response text received from the server is converted into an elderly-sounding voice, which generates a natural response that is easy for fraudsters to recognize and sends it to the fraudster.

[0525] User (system administrator) processing

[0526] System Monitoring:

[0527] Users can check whether the system is operating normally through dashboards and monitoring tools that monitor the overall system status, and if an abnormality occurs, they can take prompt action.

[0528] System optimization:

[0529] Based on the collected data and system performance data, users can adjust system parameters as needed, including retraining AI models as new fraud techniques are discovered.

[0530] Specific examples

[0531] To give a concrete scenario, consider a situation where a scammer calls and says, "Your bank account is at risk. We need your immediate attention!"

[0532] Terminal handling:

[0533] The device automatically receives the call and converts the recorded voice into text in real time, which is then sent to the server.

[0534] Server Action:

[0535] The server analyzes the received text, determines it is likely a scam, and then generates an elderly-sounding response to the device: "Sorry, what do I do?"

[0536] Terminal handling:

[0537] The device converts the received response text into speech and responds to the fraudster, then records the fraudster's response again and sends it to a server to collect the fraudster's information.

[0538] In this way, the server, terminal, and user work together to interact with fraudsters, and collect, analyze, and share the information necessary to identify them, thereby realizing a system that prevents fraudulent acts before they occur.

[0539] The processing flow will be explained below.

[0540] Step 1:

[0541] Data collection (server processing)

[0542] The server periodically collects fraud data from internet fraud databases and other sources. This data includes information about fraudsters' tactics, communication methods, victim details, location and time of occurrence. This data is stored in an internal database.

[0543] Step 2:

[0544] AI model training (server processing)

[0545] The server uses the collected data to retrain the artificial intelligence model to learn fraud patterns, which is then used to identify fraud and generate responses the next time.

[0546] Step 3:

[0547] Waiting for and receiving calls (terminal processing)

[0548] The device will always be available to receive calls using an unused phone number. If a scammer calls, the device will automatically answer the call and start the conversation.

[0549] Step 4:

[0550] Speech recognition and text conversion (device processing)

[0551] The device records the fraudster's voice in real time and converts it into text using speech recognition technology, which is then sent to a server.

[0552] Step 5:

[0553] Fraud detection (server processing)

[0554] The server analyzes the text data sent from the device and uses an AI model to determine whether there is a high possibility of fraud. If it is determined that there is a high possibility of fraud, it proceeds to the next step.

[0555] Step 6:

[0556] Response generation (server processing)

[0557] The server generates an appropriate elderly-style response based on the fraudulent scheme, which is used to continue the dialogue with the fraudster. The generated response text is sent to the device.

[0558] Step 7:

[0559] Audio conversion and transmission (terminal processing)

[0560] The device receives the response text from the server and converts it into an elderly-sounding voice using speech synthesis technology. The conversation continues when the device responds to the fraudster with the converted voice.

[0561] Step 8:

[0562] Information collection (server processing)

[0563] The server continuously analyzes the information obtained from interactions with the fraudsters and collects their characteristics and specific information (phone number, methods, etc.) in a database.

[0564] Step 9:

[0565] Information sharing (server processing)

[0566] The server collects and analyzes information on fraudsters and shares it with relevant agencies such as the police in real time, which contributes to the identification and arrest of fraudsters.

[0567] Step 10:

[0568] System monitoring (user actions)

[0569] Users regularly check the dashboard to monitor the system's operating status, and if an abnormality or error occurs, they can quickly take action.

[0570] Step 11:

[0571] System optimization (user processing)

[0572] Users analyze system performance data and adjust configuration parameters as needed, as well as retrain AI models to adapt to new fraudulent schemes.

[0573] Example 1

[0574] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0575] Fraudulent phone calls have become a social problem, with the methods becoming more sophisticated and the damage caused by them increasing. Fraud targeting the elderly in particular often leads to serious consequences. Furthermore, fraudsters often use international networks, making it difficult to deal with them using conventional methods. For this reason, there is a need for early detection and response to fraudulent phone calls, but current systems have difficulty determining fraud in real time, gathering information through dialogue, and identifying fraudsters. This article presents a system to solve this problem.

[0576] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0577] In this invention, the server includes means for collecting data on fraud, means for training an artificial intelligence model that learns fraud patterns based on the collected data, means for receiving calls from fraudsters and converting the content of the calls into text data, means for determining the possibility of fraud using the converted text data, means for generating a dialogue response using natural language generation technology if there is a possibility of fraud, means for converting the generated dialogue response into voice and replying to the fraudster, and means for organizing the characteristics and information of the fraudster and sharing that information with the relevant authorities in real time. This enables immediate identification of fraudulent calls, information collection through real-time dialogue, and rapid information sharing with the relevant authorities.

[0578] "Fraud Data" refers to information relating to fraudulent activity, such as the modus operandi, means of communication, victim details, location and time of occurrence.

[0579] "Artificial intelligence model" refers to a program that uses machine learning algorithms to learn patterns of fraudulent activity and determine the likelihood of fraud.

[0580] "Character data" refers to data converted from audio into text format.

[0581] "Natural language generation technology" refers to technology that uses AI to generate natural human conversational sentences.

[0582] "Dialogue Response" refers to text or audio generated to respond in a conversation with the fraudster.

[0583] "Relevant Authorities" refers to the police and other legal authorities that monitor and investigate fraudulent activities.

[0584] "Real-time sharing" refers to immediately transmitting information about fraudulent activity to relevant authorities.

[0585] MODE FOR CARRYING OUT THE INVENTION

[0586] The system of this invention is a system that receives fraudulent phone calls, analyzes the content of the calls, and reports them to the relevant authorities to prevent fraud. Below, we will explain in detail the roles and specific processes of the server, terminal, and user.

[0587] Server Processing

[0588] Data collection:

[0589] The server periodically collects data from fraud databases and other sources on the internet, including information on the modus operandi, means of communication, victim details, location and time of occurrence, using information providers such as "Fraud Risk Database" and "ScanWatch."

[0590] Training the AI ​​model:

[0591] The server uses the collected data to train artificial intelligence models using machine learning frameworks such as TensorFlow and PyTorch, which are then used to build algorithms that learn fraud patterns and determine likelihood of fraud.

[0592] Fraud judgment:

[0593] The server receives the text data of the incoming call sent from the device. This data is analyzed using natural language processing tools such as NLTK and spacy. The AI ​​model determines the likelihood of fraud and, if there are signs of fraud, proceeds with further processing.

[0594] Response generation:

[0595] If fraud is deemed likely, the server uses a generative AI model such as GPT-4 to generate a natural-sounding elderly-sounding response that continues the dialogue with the fraudster and elicits further information.

[0596] Data sharing:

[0597] The server collects the characteristics and identifying information of fraudsters and shares it with the relevant authorities (police, etc.) in real time. This process uses a "real-time data sharing API" and a "RESTful API" to enable rapid data transmission.

[0598] Terminal handling

[0599] Receiving a call:

[0600] The device uses VoIP technology to keep unused phone numbers constantly available for calls, and automatically receives and records calls from scammers.

[0601] Voice Recognition:

[0602] The recorded voice is converted into text data in real time using Google Cloud Speech-to-Text technology, which is then sent to a server.

[0603] Audio conversion and transmission:

[0604] The response text received from the server is converted into an elderly-sounding voice using a speech synthesis service such as Amazon Polly, which is then sent back to the scammer to continue the conversation.

[0605] User (system administrator) processing

[0606] System Monitoring:

[0607] Users monitor the overall operation of the system through dashboards and monitoring tools such as Grafana and Nagios, and if an abnormality occurs in the system, users can take prompt action.

[0608] System optimization:

[0609] Users can adjust system parameters based on collected data and system performance data, and retrain the AI ​​model using Jupyter Notebook or Google Colab as new fraud schemes are discovered.

[0610] Specific examples

[0611] To give a concrete scenario, consider a situation where a scammer calls and says, "Your bank account is at risk. We need your immediate attention!"

[0612] 1. Your device will automatically receive this call and record the audio.

[0613] 2. The recorded audio is converted into text in real time using Google Cloud Speech-to-Text and sent to the server.

[0614] 3. The server analyzes the received text and uses an AI model to determine whether it is likely to be fraudulent.

[0615] 4. Next, generate an elderly-sounding response, "Sorry, what can I do?", and send it to the device.

[0616] 5. The device converts the received text into speech using Amazon Polly and responds to the scammer.

[0617] 6. The scammer's responses are recorded again and sent to the server to collect the scammer's information.

[0618] This process allows for the collection of necessary information while maintaining a dialogue with the scammer, enabling prompt reporting to the appropriate authorities. An example prompt is, "A scammer has called an elderly person to inform them of a risk to their bank account. Please generate a natural response that sounds like an elderly person."

[0619] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0620] Step 1:

[0621] Input: The server collects fraud data from fraud databases and other sources on the Internet.

[0622] How it works: The server periodically retrieves data using designated APIs and web scraping techniques, specifically using a "fraud risk database" and "information collection tools."

[0623] Output: The collected data is stored in a database on the server.

[0624] Step 2:

[0625] Input: Collected fraud data

[0626] How it works: The server preprocesses the collected data, which includes cleaning it, imputing missing values, and normalizing it. It then trains an artificial intelligence model using machine learning frameworks like TensorFlow and PyTorch.

[0627] Output: A trained AI model is generated.

[0628] Step 3:

[0629] Input: Call from scammer

[0630] How it works: The device uses VoIP technology to receive calls from scammers. If a call comes in from a scammer, the device automatically receives and records the call.

[0631] Output: Recorded audio data is obtained.

[0632] Step 4:

[0633] Input: Recorded audio data

[0634] How it works: The device uses Google Cloud Speech-to-Text to convert recorded audio data into text data in real time.

[0635] Output: The converted character data is generated and sent to the server.

[0636] Step 5:

[0637] Input: Character data sent from the terminal

[0638] How it works: The server uses NLTK and spacy to process the received text data and uses AI models to determine the likelihood of fraud. For example, a sentence like "Your bank account is at risk" would be considered a sign of fraud.

[0639] Output: The result is that there is a high possibility of fraud.

[0640] Step 6:

[0641] Input: Character data when it is determined that there is a high possibility of fraud

[0642] How it works: The server uses a generative AI model such as GPT-4 to generate natural-sounding elderly-sounding responses, such as "I'm sorry, what should I do?"

[0643] Output: The generated dialogue response text is generated and sent to the terminal.

[0644] Step 7:

[0645] Input: Dialogue response text sent from the server

[0646] How it works: The device uses a speech synthesis service such as Amazon Polly to convert dialogue response text into an elderly-sounding voice.

[0647] Output: The generated audio data is obtained and sent to the impostor.

[0648] Step 8:

[0649] Input: Scammer's voice response data

[0650] How it works: The scammer's responses are recorded again and converted to text using Google Cloud Speech-to-Text, which is then sent back to the server.

[0651] Output: The scammer's response text data is obtained.

[0652] Step 9:

[0653] Input: Scammer's response text data sent to the server

[0654] How it works: The server compiles the characteristics and identifying information of fraudsters and shares them in real time with relevant authorities (such as the police) using a "real-time data sharing API" or "RESTful API."

[0655] Output: Information about the fraudster that has been shared with the appropriate authorities.

[0656] (Application example 1)

[0657] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0658] Telephone scams by fraudsters are becoming more sophisticated every day, resulting in many victims. Elderly people are particularly vulnerable to fraud, making countermeasures urgently needed. However, current security systems have difficulty detecting fraud in real time and taking countermeasures. There is also the risk that victims may have to confront the fraudsters themselves. Therefore, there is a need to automate the entire process of receiving and identifying fraudulent phone calls, generating dialogue responses, collecting information, and reporting to the relevant authorities, in order to prevent damage before it occurs.

[0659] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0660] In this invention, the server includes: means for collecting data on fraud; means for training an artificial intelligence model to learn fraud patterns based on the collected data; means for receiving calls from fraudsters and converting the content of the calls into text; means for determining the possibility of fraud using the converted text; means for generating a dialogue response if there is a possibility of fraud; means for converting the generated dialogue response into voice and responding to the fraudster; means for collecting information on the fraudster and sharing the information with relevant institutions; means for the smart device to receive calls and convert the voice into text in real time; and means for issuing a warning to the user based on the determination result and generating an elderly-style response. This automates the process from receiving fraudulent calls to determining whether they are fraudulent, generating a dialogue response, collecting information, and reporting to relevant institutions, making it possible to protect victims, especially elderly people, from fraud.

[0661] "Fraud Data" means information relating to fraudulent activity, including data about the fraudster's methods, means of communication, details of the victim, and the location and time of the fraud.

[0662] An "artificial intelligence model" is an algorithm that learns patterns from large amounts of data and determines the likelihood of fraud based on new data.

[0663] "Means for converting to text" refers to the process of converting voice data into text data using voice recognition technology.

[0664] The "means for determining the likelihood of fraud" is an algorithm or machine learning model that analyzes the collected text data and determines whether it constitutes fraudulent activity.

[0665] The "means for generating a dialogue response" is a process that automatically creates an appropriate response to continue dialogue with the fraudster when it is determined that there is a high possibility of fraud.

[0666] The "means for converting to speech" is a speech synthesis technology for converting the generated text dialogue response into speech output.

[0667] "Means for sharing with relevant authorities" refers to a data transmission function that allows information about fraudsters to be reported to relevant authorities and security agencies.

[0668] "Means by which a smart device receives a call" refers to the process by which a smartphone or other mobile device receives a call from a scammer and initiates processing.

[0669] "Means for converting to text in real time" refers to speech recognition technology that instantly converts received speech into text data.

[0670] The "means of issuing a warning to the user and generating an elderly-sounding response" is a mechanism that, if it is determined that there is a high possibility of fraud, displays a fraud warning to the user and generates a response that allows the user to continue a natural dialogue with the fraudster.

[0671] The system of this invention provides a specific process for automatically detecting fraudulent calls and taking countermeasures. This system is mainly composed of three parts: a server, a terminal, and a user.

[0672] Server Processing

[0673] 1. Data Collection:

[0674] The server periodically collects fraud data from fraud databases and other sources on the internet, including information about fraudsters' tactics, methods of communication, victim details, and location and time of occurrence.

[0675] 2. Training the AI ​​model:

[0676] The server trains artificial intelligence models to identify and learn from the collected data, which are then used to update criteria and algorithms for determining potential fraud.

[0677] 3. Fraud determination:

[0678] The server analyzes the text data sent from the device during the incoming call, and uses that data to determine the possibility of fraud using an AI model.

[0679] 4. Response Generation:

[0680] If fraud is determined to be likely, the server generates natural responses in an elderly voice that serve to continue the dialogue with the fraudster and elicit further information.

[0681] 5. Data Sharing:

[0682] The characteristics and identifying information of fraudsters are collected, collated, and shared with the relevant authorities (police, etc.) in real time, which will contribute to the identification and arrest of fraudsters.

[0683] Terminal handling

[0684] 1. Receiving a call:

[0685] The device will always be available to receive calls using an unused phone number, and if a call comes in from a scammer, it will automatically receive and answer the call.

[0686] 2. Speech Recognition:

[0687] The conversation with the scammer is recorded in real time and converted into text using speech recognition technology. This converted text data is sent to a server, which allows the received voice to be instantly converted into text. The hardware used could be a smartphone or mobile device, and the software could be Google Speech-to-Text API or IBM Watson Speech to Text.

[0688] 3. Audio conversion and transmission:

[0689] The response text received from the server is converted into an elderly-sounding voice, which generates a natural response that is easy for fraudsters to recognize and sends it to them. The voice synthesis technology used is gTTS (Google Text-to-Speech).

[0690] User (system administrator) processing

[0691] 1. System Monitoring:

[0692] Users can check whether the system is operating normally through dashboards and monitoring tools that monitor the overall system status, and if an abnormality occurs, they can take prompt action.

[0693] 2. System optimization:

[0694] Users can adjust system parameters based on collected data and system performance data, including retraining AI models as new fraud techniques are discovered.

[0695] Specific examples

[0696] For example, consider the case where a fraudster calls and says, "Your bank account is at risk. We need your immediate attention!" First, the device automatically receives the call and converts the recorded voice into text in real time. This converted text is sent to a server, which analyzes the received text and determines that it is likely a fraud. Next, the device generates an elderly-sounding response such as "I'm sorry, what should I do?" and sends it to the device. The device then converts the received response text into voice and replies to the fraudster. The fraudster's response is recorded again and further information is sent to the server. This series of steps prevents fraud before it occurs.

[0697] Prompt Sentence Examples

[0698] This is an example of actual input to a generative AI model.

[0699] "Please recognize the voice of this call and determine if it is likely a scam. If it is, please display a warning and generate an elderly-sounding response to continue the conversation with the scammer."

[0700] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0701] Step 1:

[0702] Receiving a call

[0703] The terminal always keeps itself available to receive calls using an unused phone number. When a call from a scammer comes in, it automatically answers the call and starts recording. At this point, the input is the audio data of the received call, and the output is the recorded audio data.

[0704] Step 2:

[0705] Voice Recognition

[0706] The device converts the recorded voice data into text in real time using speech recognition technology. The input is the recorded voice data, and the output is text data. This process uses the Google Speech-to-Text API and IBM Watson Speech to Text.

[0707] Step 3:

[0708] Fraud judgment

[0709] The server receives the text data sent from the device and uses an artificial intelligence model to determine the likelihood of fraud. The input is the text data sent from the device, and the output is the determination result regarding the likelihood of fraud. A machine learning algorithm is used for data analysis.

[0710] Step 4:

[0711] Warning notice

[0712] If the server determines that there is a high possibility of fraud, it sends a warning to the terminal. The input is the result of the fraud determination, and the output is a warning to the user. This warning is displayed to the user through the terminal interface.

[0713] Step 5:

[0714] Response Generation

[0715] If the server determines that fraud is likely, it generates a natural response in the style of an elderly person. This generated response serves to continue the dialogue with the fraudster and elicit further information. The input is the result of the fraud detection, and the output is a response sentence in the style of an elderly person. A generative AI model is used to generate this response.

[0716] Step 6:

[0717] Voice conversion and transmission

[0718] The device converts the response text received from the server into an elderly-sounding voice. The voice synthesis technology used is gTTS (Google Text-to-Speech). The input is the generated response text, and the output is voice data. The converted voice is then sent back to the fraudster.

[0719] Step 7:

[0720] Scammers' information gathering and sharing

[0721] The device transmits the recorded voice data during the ongoing conversation with the fraudster to the server, which analyzes it, extracts the fraudster's characteristic information, and shares it with relevant organizations. The input is the recorded voice data, and the output is the analyzed characteristic information of the fraudster and the data to be shared with relevant organizations.

[0722] This series of steps will automate the entire process, from receiving fraudulent calls to identifying them, generating dialogue responses, collecting information, and reporting them to relevant authorities.

[0723] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0724] This invention is a system designed to receive and analyze calls from fraudsters and strengthen fraud prevention measures, and by combining it with an emotion engine to recognize user emotions, it further enhances accuracy. This system consists of three main components: a server, a terminal, and a user. The process and operation of each component are explained in detail below.

[0725] Server Processing

[0726] Data collection

[0727] The server periodically collects fraud data from online fraud databases and other sources. This data includes information about fraudsters' tactics, communication methods, victim details, location and time of occurrence. The collected data is stored in the server's internal database.

[0728] Training an AI model

[0729] Based on the collected data, the server trains an artificial intelligence model to learn fraud patterns, which includes criteria and algorithms to determine the likelihood of fraud.

[0730] Fraud judgment

[0731] The server analyzes the text data of the incoming call sent from the device and uses an AI model to determine the possibility of fraud. If it determines that there is a high possibility of fraud, it proceeds to the next step.

[0732] Response Generation

[0733] If fraud is determined to be likely, the server generates a natural-sounding response in the style of an elderly person, which is used to continue the dialogue with the fraudster and extract further information. The generated response text is then sent to the device.

[0734] Data Sharing

[0735] Collect and organize information about the characteristics and identity of fraudsters, and share it with the relevant authorities (police, etc.) in real time. This information sharing will contribute to the identification and arrest of fraudsters.

[0736] Terminal handling

[0737] Receiving a call

[0738] The device will use an unused phone number and will always be available to receive calls. If a scammer calls, the device will automatically receive and answer the call.

[0739] Voice Recognition

[0740] The device records the fraudster's voice in real time and converts it into text using voice recognition technology, which is then sent to a server.

[0741] Voice conversion and transmission

[0742] The response text received from the server is converted into an elderly-sounding voice, and the conversation continues by replying to the fraudster with the converted voice.

[0743] emotion recognition

[0744] The emotion engine installed on the device analyzes the user's emotions during the conversation with the scammer. For example, emotions such as anger, anxiety, and tension can be detected from the scammer's voice. This emotional information is sent to the server and reflected in the response generation.

[0745] User (system administrator) processing

[0746] System Monitoring

[0747] Users can check the dashboard to monitor the overall system status and quickly respond to any abnormalities or errors that occur.

[0748] System Optimization

[0749] Users can adjust system parameters based on collected data and system performance data, and retrain AI models to adapt to new fraudulent schemes.

[0750] Specific examples

[0751] As a specific scenario, let's consider a situation where a scammer calls and says, "Your bank account has been compromised. We need your immediate attention."

[0752] Terminal handling:

[0753] The device automatically receives the call and records the audio, which is then converted into text using voice recognition technology and sent to a server.

[0754] Server Action:

[0755] The server analyzes the received text, determines it is likely a scam, and then generates an elderly-sounding response to the device: "Sorry, what do I do?"

[0756] Terminal handling:

[0757] The device converts the received response text into speech and responds to the fraudster, while simultaneously analyzing the emotions in the fraudster's voice and sending the emotional information to the server.

[0758] Server Action:

[0759] The server uses this emotional information to generate an appropriate response that reflects the fraudster's emotional state when generating the next response. For example, if the fraudster is feeling anxious, the server can use this to devise a response that will further encourage the fraudulent tactics.

[0760] In this way, the server, terminal, and user work together to communicate with fraudsters in order to prevent fraud before it happens. Furthermore, by combining it with an emotion engine, the system can analyze the emotions of fraudsters, enabling more effective dialogue and information gathering.

[0761] The processing flow will be explained below.

[0762] Step 1:

[0763] Data collection (server processing)

[0764] The server periodically collects fraud data from fraud databases on the Internet and other sources. This data includes information about fraudsters' tactics, communication methods, victim details, location and time of occurrence. This collected data is stored in an internal database.

[0765] Step 2:

[0766] AI model training (server processing)

[0767] The server uses the collected data to retrain its artificial intelligence model, which learns fraud patterns, allowing it to adapt to the latest fraud techniques. The trained AI model is then used to identify fraud and generate responses for the next transaction.

[0768] Step 3:

[0769] Waiting for and receiving calls (terminal processing)

[0770] The device will use an unused phone number and will always be available to receive calls. If a scammer calls, the device will automatically answer the call and initiate the call.

[0771] Step 4:

[0772] Speech recognition and text conversion (device processing)

[0773] The device records the fraudster's voice in real time and converts this voice data into text using speech recognition technology, which is then sent to a server.

[0774] Step 5:

[0775] Fraud detection (server processing)

[0776] The server analyzes the text data sent from the device and uses an AI model to determine the likelihood of fraud. If it determines that there is a high possibility of fraud, it proceeds to the next step.

[0777] Step 6:

[0778] Emotion recognition (device processing)

[0779] An emotion engine on the device analyzes the scammer's voice and determines their emotional state (e.g., anger, impatience, calm, etc.) This emotion data is sent to the server for use in generating subsequent responses.

[0780] Step 7:

[0781] Response generation (server processing)

[0782] The server generates appropriate responses in the style of an elderly person based on the fraudster's tactics and emotional data. The responses are intended to continue the dialogue with the fraudster and elicit further information. The generated response text is then sent to the device.

[0783] Step 8:

[0784] Audio conversion and transmission (terminal processing)

[0785] The device uses speech synthesis technology to convert the response text received from the server into an elderly-sounding voice, which is then sent back to the fraudster, continuing the dialogue.

[0786] Step 9:

[0787] Information collection (server processing)

[0788] The server analyzes the information collected through interactions with the fraudsters and stores their characteristics (phone number, tactics, emotional state, etc.) in a comprehensive database.

[0789] Step 10:

[0790] Information sharing (server processing)

[0791] The server then shares the collected and analyzed information on fraudsters with the police and other relevant authorities in real time. This information sharing process facilitates the identification and arrest of fraudsters.

[0792] Step 11:

[0793] System monitoring (user actions)

[0794] Users can use a dashboard to monitor the overall system operation status and respond quickly to any errors or abnormalities that occur. Daily monitoring maintains the reliability and stability of the system.

[0795] Step 12:

[0796] System optimization (user processing)

[0797] Users can analyze the system's performance data and adjust configuration parameters as needed, including instructing the AI ​​model to retrain to adapt to new fraudulent schemes.

[0798] In a specific scenario, if a fraudster calls and says, "Your bank account is at risk. Immediate action is required," the system works as follows: The device receives the call and uses voice recognition technology to convert the content into text and send it to the server. The server analyzes the text, determines whether it is likely to be fraudulent, generates a response text, and sends it to the device. The device then converts the response text into speech and replies to the fraudster. During this process, the emotion engine analyzes the fraudster's emotions, and this data is reflected in the generation of the next response.

[0799] Example 2

[0800] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0801] In recent years, fraud has become more sophisticated and diverse, making it difficult to respond to conventional countermeasures. Telephone fraud targeting the elderly has been increasing in particular, and effective countermeasures are needed. Systems are needed that not only prevent fraud from occurring, but also collect and share information in real time to identify and arrest fraudsters. Furthermore, there is a need to expose fraudulent activities by analyzing fraudsters' emotions and engaging in more effective dialogue.

[0802] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting data related to fraud, means for training an artificial intelligence model to learn fraud patterns based on the collected data, means for receiving communications from fraudsters and converting the content of the communications into text, means for determining the possibility of fraud using the converted text, means for generating a dialogue response if there is a possibility of fraud, means for converting the generated dialogue response into voice and replying to the fraudster, means for analyzing the characteristics of the fraudster and obtaining emotional information, and means for sharing information about the fraudster with relevant agencies. This improves the accuracy of fraud countermeasures, prevents fraudulent acts, and enables the collection and sharing of information to identify and arrest fraudsters in real time.

[0803] "Fraud Data" means information relating to fraudulent activity, including information such as the fraudulent method, means of communication, details of the victim, and the location and time of the occurrence.

[0804] An "artificial intelligence model" refers to an algorithm that learns from data and automatically performs specific tasks.

[0805] "Communications" refers to the means of communication used by scammers, such as voice or messages.

[0806] "Means for converting to text" refers to speech recognition technology for converting voice or messages into text data.

[0807] "Means for determining likelihood of fraud" refers to algorithms or models that analyze the converted text data and assess the risk of fraud.

[0808] "Means for generating dialogue responses" refers to an algorithm or generative model for automatically generating appropriate responses to continue the dialogue with the fraudster.

[0809] "Means for converting to speech" refers to speech synthesis technology for playing the generated text dialogue response as speech.

[0810] "Means of obtaining emotional information" refers to technology that analyzes emotions from the fraudster's voice and messages and identifies emotional states such as anger or impatience.

[0811] "Means of sharing information with relevant agencies" refers to communication methods and interfaces for providing information about fraudsters to relevant agencies, such as the police, in real time.

[0812] This invention is a system designed to prevent fraud by receiving calls from fraudsters, analyzing them, and strengthening countermeasures. The system consists of three main components: a server, a terminal, and a user, which work together.

[0813] Server Processing

[0814] Data collection

[0815] The server collects fraud data from online fraud databases and other sources, including fraudster tactics, communication methods, victim details, location and time of occurrence. Specifically, it retrieves data from the ScamWatch and FraudLog databases using HTTP requests and stores it in the server's internal database.

[0816] Training an AI model

[0817] The collected data is preprocessed and an artificial intelligence model is trained. For example, TensorFlow or PyTorch is used to learn fraud patterns. Specific features such as specific methods and time periods are incorporated into the algorithm to improve the accuracy of fraud detection.

[0818] Analysis of incoming calls and fraud detection

[0819] The server receives the text data of the incoming call sent from the device and analyzes it using an AI model. The analysis uses keywords and phrases contained in the text data to determine the possibility of fraud. For example, phrases such as "bank account," "risk," and "action required" are determined to indicate a high risk of fraud.

[0820] Response Generation

[0821] If fraud is deemed likely, the server uses a generative AI model (e.g., OpenAI GPT-4) to generate natural-sounding responses in the style of an elderly person, designed to continue the dialogue with the fraudster and elicit further information, such as "I'm sorry, what should I do?"

[0822] Data Sharing

[0823] The characteristics and identifying information of fraudsters are collated and shared with the relevant authorities (e.g., police) in real time. This information is sent via API to expedite response.

[0824] Terminal handling

[0825] Receiving a call

[0826] The device is set to an unused phone number and is always ready to receive calls. When a call comes in from a scammer, it automatically receives the call and prepares to record it. Specifically, it uses Twilio's call receiving API.

[0827] Speech recognition and text conversion

[0828] The device records the fraudster's voice in real time and converts it into text data using Google Cloud Speech-to-Text. The recorded audio stream is sent to an API, which outputs the text data, which is then sent to the server.

[0829] Voice conversion and transmission

[0830] The response text received from the server is converted into an elderly-sounding voice. Specifically, speech synthesis is performed using Amazon Polly or Google Text-to-Speech. The converted voice is then sent back to the fraudster, and the dialogue continues.

[0831] emotion recognition

[0832] The device uses the Microsoft Azure Emotion API to analyze emotional information from the fraudster's voice, detecting emotions such as anger, worry, and tension, and sending this emotional information to a server.

[0833] User (system administrator) processing

[0834] System Monitoring

[0835] Users use Grafana and Kibana to monitor the overall system status. If an abnormality or error occurs, they receive notifications and can respond quickly. They can understand the situation by checking log data and real-time notifications.

[0836] System Optimization

[0837] Users can adjust system parameters based on collected data and system performance data, and retrain the AI ​​model as needed to update the system to adapt to new fraudulent schemes.

[0838] Specific examples

[0839] As a specific scenario, let's consider a situation where a scammer calls and says, "Your bank account has been compromised. We need your immediate attention."

[0840] Terminal handling

[0841] The device automatically receives the call using Twilio, then converts the fraudster's speech into text in real time using Google Cloud Speech-to-Text, which then sends the text data to the server via an RTP stream.

[0842] Server Processing

[0843] The server receives the text data, "Your bank account is at risk. Immediate action is required." and analyzes it using an AI model. If it determines there is a high possibility of fraud, it uses OpenAI's GPT-4 to generate an elderly-sounding response, "I'm sorry, what should I do?", and sends it to the device.

[0844] Terminal handling

[0845] The device uses Amazon Polly to convert the received response text into an elderly-sounding voice and responds to the fraudster. At the same time, it uses Microsoft Azure's Emotion API to analyze emotional information (such as impatience or tension) from the fraudster's voice and sends it to the server.

[0846] Server Processing

[0847] The server will use this emotional information to generate the next response. For example, if it determines that the fraudster is anxious, it can generate a question like, "I'd like to confirm again, is your account really at risk?" This can further refine the fraudulent scheme.

[0848] Example prompts to input to the generative AI model

[0849] "Please translate the following sentence into a natural response from a senior citizen: 'Your bank account is at risk. Action is required immediately.'"

[0850] This prompt can be used to automatically generate an appropriate response.

[0851] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0852] Server Processing

[0853] Step 1: Data collection

[0854] The server collects fraud data from fraud databases and other sources on the internet. Specifically, it uses HTTP requests to retrieve information from the database about the fraud scheme, communication method, victim details, location and time of the incident. The input is the database API endpoint, and the output is detailed data about the fraud. The retrieved data is stored in the server's internal database.

[0855] Step 2: Training the AI ​​model

[0856] The server preprocesses the collected data and trains an artificial intelligence model. Specifically, it uses TensorFlow and PyTorch to learn fraud patterns. The input is the preprocessed dataset, and the output is a trained AI model. Features such as specific methods and time periods are reflected in the algorithm to improve the model's accuracy.

[0857] Step 3: Analyze incoming calls and identify fraud

[0858] The server receives the text data of the incoming call sent from the device and analyzes it using an AI model. The input is text data, and the output is a judgment result indicating the possibility of fraud. In this process, keywords and phrases contained in the text are analyzed to assess the risk of fraud.

[0859] Step 4: Response Generation

[0860] If the server determines that there is a high possibility of fraud, it uses a generative AI model to generate a natural-sounding response in the style of an elderly person. The input is the judgment result and the fraudster's text data, and the output is the generated response text. For example, it generates a response such as, "I'm sorry, what should I do?"

[0861] Step 5: Data sharing

[0862] The server compiles the characteristics and identifying information of fraudsters and shares it with the relevant authorities in real time. The input is detailed information about the fraudsters, and the output is the information sent to the relevant authorities. The information is sent quickly through the API, contributing to the identification and arrest of fraudsters.

[0863] Terminal handling

[0864] Step 1: Receiving a call

[0865] The terminal is set to an unused phone number and is always ready to receive calls. The input is a call from a fraudster, and the output is the received voice data. Specifically, it uses Twilio's call receiving API to automatically receive calls.

[0866] Step 2: Speech recognition and text conversion

[0867] The device records the fraudster's voice in real time and converts it into text data using Google Cloud Speech-to-Text. The input is audio data and the output is text data. The recorded audio stream is sent to the API, and the converted text is sent to the server.

[0868] Step 3: Convert and send

[0869] The device converts the response text received from the server into an elderly-sounding voice. The input is the response text, and the output is synthesized voice data. Specifically, Amazon Polly and Google Text-to-Speech are used to generate the voice and respond to the scammer.

[0870] Step 4: Emotion Recognition

[0871] An emotion engine implemented on the device is used to analyze emotions from the fraudster's voice. The input is the fraudster's voice data, and the output is emotional information (e.g., anger, tension, etc.). The emotional information is sent to the server and reflected in the generation of the next response.

[0872] User (system administrator) processing

[0873] Step 1: System monitoring

[0874] Users check a dashboard to monitor the overall system operation status. The input is real-time system log data, and the output is the monitoring results. Specifically, Grafana and Kibana are used to monitor and receive notifications for abnormalities and errors.

[0875] Step 2: System optimization

[0876] Users adjust the system parameters based on collected data and system performance data. The input is performance data and information on new fraud schemes, and the output is optimized system settings. If necessary, the AI ​​model can also be retrained.

[0877] (Application example 2)

[0878] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0879] The number of fraudulent phone calls, often involving sophisticated tactics by scammers, is steadily increasing, with elderly people being particularly vulnerable to these attacks. Current systems for combating fraudulent phone calls often struggle to identify these calls, resulting in delayed effective responses. Furthermore, they lack the means to analyze the emotions and psychological state of fraudsters, making it difficult to effectively deter and collect information from them. This can result in investigative authorities being unable to respond quickly, potentially exacerbating the damage. Therefore, there is a need for a system that can improve the accuracy and effectiveness of countermeasures against fraudulent phone calls by recognizing the emotions of fraudsters and generating responses that sound like an elderly person.

[0880] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0881] In this invention, the server includes means for collecting data on fraud, means for training an artificial intelligence model that learns fraud patterns based on the collected data, means for determining the possibility of fraud using the converted text, means for generating an elderly-sounding dialogue response if there is a possibility of fraud, means for converting the generated dialogue response into voice and replying to the fraudster, means for analyzing emotions from the fraudster's voice, and means for collecting information on the fraudster and sharing it with relevant institutions. This makes it possible to provide an appropriate response according to the fraudster's emotional state, effectively continue the dialogue to collect information, and ultimately prevent fraud before it occurs.

[0882] "Fraud Data" means any information relating to fraudulent activity, including the method of fraud, means of communication, details of the victim, location and time of the occurrence.

[0883] An "artificial intelligence model" refers to a computer model that can learn from large amounts of data and identify and analyze specific patterns and features.

[0884] "Convert to text" refers to the process of converting audio data into text data.

[0885] "Means for determining the likelihood of fraud" refers to the process of using artificial intelligence models or other algorithms to analyze and determine whether fraud is suspected.

[0886] "Elderly-style dialogue responses" refers to text and voice responses generated to make it appear as if an elderly person is having a conversation.

[0887] "Convert to voice" refers to the process of converting text data into voice data.

[0888] "Means for analyzing emotions from fraudsters' voices" refers to technology for identifying the emotional state (e.g., anger, impatience, tension, etc.) of fraudsters from their vocalizations.

[0889] "Fraudster information" refers to any information that can be used to identify a fraudster, including voice characteristics, phone number, words used, and conversation content.

[0890] "Means of sharing with relevant agencies" refers to the process for providing collected and analyzed fraud information to investigative agencies such as the police and monitoring agencies.

[0891] The present invention is a system for dealing with fraudulent calls from fraudsters, and is composed of a server, a terminal, and a user. The processing of the entire system will be described in detail below.

[0892] Server Processing

[0893] Data collection

[0894] The server periodically collects fraud data from online fraud databases and other sources. This data includes information about the fraud scheme, means of communication, victim details, location and time of occurrence. The collected data is stored in the server's internal database.

[0895] Training an AI model

[0896] Based on the collected data, the server trains an artificial intelligence model to learn fraud patterns, specifically by using pattern recognition techniques and machine learning algorithms to extract and learn the characteristics of suspicious communications.

[0897] Fraud judgment

[0898] The server analyzes the text data of the incoming call sent from the terminal and uses an artificial intelligence model to determine the possibility of fraud. If the result of this determination indicates a high possibility of fraud, the system proceeds to the next step.

[0899] Response Generation

[0900] If the server determines that fraud is likely, it generates a natural response in the style of an elderly person. This response is used to continue the dialogue with the fraudster and extract further information. The generated response text is sent to the device.

[0901] Data Sharing

[0902] Collecting characteristics and identifying information of fraudsters and sharing it with relevant authorities (such as the police) in real time, which will contribute to identifying and arresting fraudsters.

[0903] Terminal handling

[0904] Receiving a call

[0905] The device will always be ready to receive calls, and if a scammer calls you, it will automatically receive and answer the call.

[0906] Voice Recognition

[0907] The device records the fraudster's voice in real time and converts it into text using voice recognition technology (e.g., Google Cloud Speech-to-Text), which is then sent to a server.

[0908] Voice conversion and transmission

[0909] The response text received from the server is converted into an elderly-sounding voice. The converted voice is then sent back to the fraudster, continuing the dialogue. Tools such as Google Text-to-Speech (gTTS) are used for voice synthesis.

[0910] emotion recognition

[0911] An emotion engine (e.g., IBM Watson Tone Analyzer) installed on the device analyzes the emotions of the fraudster during the conversation. The analysis results are sent to the server and reflected in the response generation.

[0912] User (system administrator) processing

[0913] System Monitoring

[0914] Users can check the dashboard to monitor the overall system status, and respond quickly if an abnormality or error occurs.

[0915] System Optimization

[0916] Users can adjust system parameters based on collected data and system performance data, and retrain AI models to adapt to new fraudulent schemes.

[0917] Specific examples

[0918] For example, imagine a scammer calling you and saying, "Your bank account is at risk. We need your immediate attention."

[0919] 1. Terminal Processing

[0920] The device automatically receives the call and records the audio, which is then converted into text using voice recognition technology and sent to a server.

[0921] 2. Server Processing

[0922] The server analyzes the received text and determines that it is likely a scam, then generates an elderly-sounding response to the user, such as "Really? What should I do?", and sends it to the device.

[0923] 3. Terminal Processing

[0924] The device converts the received response text into speech and responds to the fraudster. It also analyzes the emotions in the fraudster's voice and sends the emotional information to the server.

[0925] 4. Server Processing

[0926] The server then uses the emotional information to generate an appropriate response that reflects the fraudster's emotional state. For example, if the fraudster is feeling anxious, the server can use this to devise a response that will further encourage the fraudulent tactics.

[0927] Example prompt sentence:

[0928] "Pretend to be a senior citizen and respond naturally to the following statement: 'Your bank account is at risk.'"

[0929] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0930] Step 1:

[0931] Receiving a call (terminal)

[0932] The device is always ready to receive calls. When a call comes in from a fraudster, it automatically answers the call and starts the conversation. The input is the call signal from the fraudster, and the output is the call audio data.

[0933] Step 2:

[0934] Voice recording and recognition (device)

[0935] The device records the scammer's voice in real time and converts the voice data into text data using voice recognition technology such as Google Cloud Speech-to-Text. The input is the recorded voice data, and the output is text data.

[0936] Step 3:

[0937] Sending text data (terminal → server)

[0938] The terminal sends the converted text data to the server. The input is the text data generated by the speech recognition technology, and the output is the text data sent to the server.

[0939] Step 4:

[0940] Fraud detection (server)

[0941] The server analyzes the received text data and uses an artificial intelligence model (generative AI model) to determine the likelihood of fraud. The input is text data, and the output is a determination result indicating the likelihood of fraud. This data is processed through data calculations and analysis within the server.

[0942] Step 5:

[0943] Generate response (server)

[0944] If the server determines that there is a high possibility of fraud, it generates a natural response in the style of an elderly person for the next dialogue step, where it uses a generative AI model and a prompt to generate an appropriate response. The input is the fraud detection result and the prompt, and the output is the generated response text.

[0945] Step 6:

[0946] Sending a response (server → terminal)

[0947] The server sends the generated response text to the terminal. The input is the generated response text, and the output is the text data sent to the terminal.

[0948] Step 7:

[0949] Voice conversion and reply (terminal)

[0950] The device converts the received response text into an elderly-sounding voice and generates an audio file using Google Text-to-Speech (gTTS), which then replies to the fraudster using this generated voice. The input is the response text, and the output is the voice response to the fraudster.

[0951] Step 8:

[0952] Emotion Recognition (Device)

[0953] The device analyzes the emotions from the fraudster's voice and sends the results to a server, using technologies such as IBM Watson Tone Analyzer. The input is the fraudster's voice data, and the output is the analyzed emotion data.

[0954] Step 9:

[0955] Receiving and responding to emotion data (server)

[0956] The server receives the emotion data sent from the device and reflects it when generating the next response based on that emotion information. This makes it possible to generate an appropriate response based on the fraudster's psychological state. The input is emotion data, and the output is the next response text based on the emotion.

[0957] Step 10:

[0958] Fraudster information collection and sharing (server)

[0959] The server organizes and collects information obtained from fraudsters and shares it with the relevant authorities (police, etc.) in real time. The input is the fraudster information obtained from the conversation, and the output is the shared information sent to the relevant authorities.

[0960] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0961] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0962] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0963] [Third embodiment]

[0964] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0965] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0966] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0967] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0968] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0969] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0970] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0971] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0972] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0973] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0974] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0975] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0976] The system of this invention is designed to prevent and expose fraudulent activities by receiving calls from fraudsters, analyzing the content of the conversation, and reporting it to the relevant authorities. Below, we will explain the specific processing and operation of this system, dividing it into three roles: server, terminal, and user.

[0977] Server Processing

[0978] Data collection:

[0979] The server periodically collects fraud data from fraud databases and other sources on the internet, including information about fraudster tactics, methods of communication, victim details, and the location and time of fraud.

[0980] Training the AI ​​model:

[0981] The server uses the collected data to train artificial intelligence models to identify and learn from fraud patterns, which are then used to update criteria and algorithms for determining potential fraud.

[0982] Fraud judgment:

[0983] The server analyzes the text data sent from the device and uses that data to determine the possibility of fraud using an AI model.

[0984] Response generation:

[0985] If fraud is determined to be likely, the server generates natural responses in an elderly voice that serve to continue the dialogue with the fraudster and elicit further information.

[0986] Data sharing:

[0987] The characteristics and identifying information of fraudsters are collected, collated, and shared with the relevant authorities (police, etc.) in real time, which will contribute to the identification and arrest of fraudsters.

[0988] Terminal handling

[0989] Receiving a call:

[0990] The device will always be available to receive calls using an unused phone number, and will automatically receive and answer calls from scammers.

[0991] Voice Recognition:

[0992] The conversation with the scammer is recorded in real time and converted into text using voice recognition technology, which is then sent to a server.

[0993] Audio conversion and transmission:

[0994] The response text received from the server is converted into an elderly-sounding voice, which generates a natural response that is easy for fraudsters to recognize and sends it to the fraudster.

[0995] User (system administrator) processing

[0996] System Monitoring:

[0997] Users can check whether the system is operating normally through dashboards and monitoring tools that monitor the overall system status, and if an abnormality occurs, they can take prompt action.

[0998] System optimization:

[0999] Based on the collected data and system performance data, users can adjust system parameters as needed, including retraining AI models as new fraud techniques are discovered.

[1000] Specific examples

[1001] To give a concrete scenario, consider a situation where a scammer calls and says, "Your bank account is at risk. We need your immediate attention!"

[1002] Terminal handling:

[1003] The device automatically receives the call and converts the recorded voice into text in real time, which is then sent to the server.

[1004] Server Action:

[1005] The server analyzes the received text, determines it is likely a scam, and then generates an elderly-sounding response to the device: "Sorry, what do I do?"

[1006] Terminal handling:

[1007] The device converts the received response text into speech and responds to the fraudster, then records the fraudster's response again and sends it to a server to collect the fraudster's information.

[1008] In this way, the server, terminal, and user work together to interact with fraudsters, and collect, analyze, and share the information necessary to identify them, thereby realizing a system that prevents fraudulent acts before they occur.

[1009] The processing flow will be explained below.

[1010] Step 1:

[1011] Data collection (server processing)

[1012] The server periodically collects fraud data from internet fraud databases and other sources. This data includes information about fraudsters' tactics, communication methods, victim details, location and time of occurrence. This data is stored in an internal database.

[1013] Step 2:

[1014] AI model training (server processing)

[1015] The server uses the collected data to retrain the artificial intelligence model to learn fraud patterns, which is then used to identify fraud and generate responses the next time.

[1016] Step 3:

[1017] Waiting for and receiving calls (terminal processing)

[1018] The device will always be available to receive calls using an unused phone number. If a scammer calls, the device will automatically answer the call and start the conversation.

[1019] Step 4:

[1020] Speech recognition and text conversion (device processing)

[1021] The device records the fraudster's voice in real time and converts it into text using speech recognition technology, which is then sent to a server.

[1022] Step 5:

[1023] Fraud detection (server processing)

[1024] The server analyzes the text data sent from the device and uses an AI model to determine whether there is a high possibility of fraud. If it is determined that there is a high possibility of fraud, it proceeds to the next step.

[1025] Step 6:

[1026] Response generation (server processing)

[1027] The server generates an appropriate elderly-style response based on the fraudulent scheme, which is used to continue the dialogue with the fraudster. The generated response text is sent to the device.

[1028] Step 7:

[1029] Audio conversion and transmission (terminal processing)

[1030] The device receives the response text from the server and converts it into an elderly-sounding voice using speech synthesis technology. The conversation continues when the device responds to the fraudster with the converted voice.

[1031] Step 8:

[1032] Information collection (server processing)

[1033] The server continuously analyzes the information obtained from interactions with the fraudsters and collects their characteristics and specific information (phone number, methods, etc.) in a database.

[1034] Step 9:

[1035] Information sharing (server processing)

[1036] The server collects and analyzes information on fraudsters and shares it with relevant agencies such as the police in real time, which contributes to the identification and arrest of fraudsters.

[1037] Step 10:

[1038] System monitoring (user actions)

[1039] Users regularly check the dashboard to monitor the system's operating status, and if an abnormality or error occurs, they can quickly take action.

[1040] Step 11:

[1041] System optimization (user processing)

[1042] Users analyze system performance data and adjust configuration parameters as needed, as well as retrain AI models to adapt to new fraudulent schemes.

[1043] Example 1

[1044] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1045] Fraudulent phone calls have become a social problem, with the methods becoming more sophisticated and the damage caused by them increasing. Fraud targeting the elderly in particular often leads to serious consequences. Furthermore, fraudsters often use international networks, making it difficult to deal with them using conventional methods. For this reason, there is a need for early detection and response to fraudulent phone calls, but current systems have difficulty determining fraud in real time, gathering information through dialogue, and identifying fraudsters. This article presents a system to solve this problem.

[1046] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1047] In this invention, the server includes means for collecting data on fraud, means for training an artificial intelligence model that learns fraud patterns based on the collected data, means for receiving calls from fraudsters and converting the content of the calls into text data, means for determining the possibility of fraud using the converted text data, means for generating a dialogue response using natural language generation technology if there is a possibility of fraud, means for converting the generated dialogue response into voice and replying to the fraudster, and means for organizing the characteristics and information of the fraudster and sharing that information with the relevant authorities in real time. This enables immediate identification of fraudulent calls, information collection through real-time dialogue, and rapid information sharing with the relevant authorities.

[1048] "Fraud Data" refers to information relating to fraudulent activity, such as the modus operandi, means of communication, victim details, location and time of occurrence.

[1049] "Artificial intelligence model" refers to a program that uses machine learning algorithms to learn patterns of fraudulent activity and determine the likelihood of fraud.

[1050] "Character data" refers to data converted from audio into text format.

[1051] "Natural language generation technology" refers to technology that uses AI to generate natural human conversational sentences.

[1052] "Dialogue Response" refers to text or audio generated to respond in a conversation with the fraudster.

[1053] "Relevant Authorities" refers to the police and other legal authorities that monitor and investigate fraudulent activities.

[1054] "Real-time sharing" refers to immediately transmitting information about fraudulent activity to relevant authorities.

[1055] MODE FOR CARRYING OUT THE INVENTION

[1056] The system of this invention is a system that receives fraudulent phone calls, analyzes the content of the calls, and reports them to the relevant authorities to prevent fraud. Below, we will explain in detail the roles and specific processes of the server, terminal, and user.

[1057] Server Processing

[1058] Data collection:

[1059] The server periodically collects data from fraud databases and other sources on the internet, including information on the modus operandi, means of communication, victim details, location and time of occurrence, using information providers such as "Fraud Risk Database" and "ScanWatch."

[1060] Training the AI ​​model:

[1061] The server uses the collected data to train artificial intelligence models using machine learning frameworks such as TensorFlow and PyTorch, which are then used to build algorithms that learn fraud patterns and determine likelihood of fraud.

[1062] Fraud judgment:

[1063] The server receives the text data of the incoming call sent from the device. This data is analyzed using natural language processing tools such as NLTK and spacy. The AI ​​model determines the likelihood of fraud and, if there are signs of fraud, proceeds with further processing.

[1064] Response generation:

[1065] If fraud is deemed likely, the server uses a generative AI model such as GPT-4 to generate a natural-sounding elderly-sounding response that continues the dialogue with the fraudster and elicits further information.

[1066] Data sharing:

[1067] The server collects the characteristics and identifying information of fraudsters and shares it with the relevant authorities (police, etc.) in real time. This process uses a "real-time data sharing API" and a "RESTful API" to enable rapid data transmission.

[1068] Terminal handling

[1069] Receiving a call:

[1070] The device uses VoIP technology to keep unused phone numbers constantly available for calls, and automatically receives and records calls from scammers.

[1071] Voice Recognition:

[1072] The recorded voice is converted into text data in real time using Google Cloud Speech-to-Text technology, which is then sent to a server.

[1073] Audio conversion and transmission:

[1074] The response text received from the server is converted into an elderly-sounding voice using a speech synthesis service such as Amazon Polly, which is then sent back to the scammer to continue the conversation.

[1075] User (system administrator) processing

[1076] System Monitoring:

[1077] Users monitor the overall operation of the system through dashboards and monitoring tools such as Grafana and Nagios, and if an abnormality occurs in the system, users can take prompt action.

[1078] System optimization:

[1079] Users can adjust system parameters based on collected data and system performance data, and retrain the AI ​​model using Jupyter Notebook or Google Colab as new fraud schemes are discovered.

[1080] Specific examples

[1081] To give a concrete scenario, consider a situation where a scammer calls and says, "Your bank account is at risk. We need your immediate attention!"

[1082] 1. Your device will automatically receive this call and record the audio.

[1083] 2. The recorded audio is converted into text in real time using Google Cloud Speech-to-Text and sent to the server.

[1084] 3. The server analyzes the received text and uses an AI model to determine whether it is likely to be fraudulent.

[1085] 4. Next, generate an elderly-sounding response, "Sorry, what can I do?", and send it to the device.

[1086] 5. The device converts the received text into speech using Amazon Polly and responds to the scammer.

[1087] 6. The scammer's responses are recorded again and sent to the server to collect the scammer's information.

[1088] This process allows for the collection of necessary information while maintaining a dialogue with the scammer, enabling prompt reporting to the appropriate authorities. An example prompt is, "A scammer has called an elderly person to inform them of a risk to their bank account. Please generate a natural response that sounds like an elderly person."

[1089] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1090] Step 1:

[1091] Input: The server collects fraud data from fraud databases and other sources on the Internet.

[1092] How it works: The server periodically retrieves data using designated APIs and web scraping techniques, specifically using a "fraud risk database" and "information collection tools."

[1093] Output: The collected data is stored in a database on the server.

[1094] Step 2:

[1095] Input: Collected fraud data

[1096] How it works: The server preprocesses the collected data, which includes cleaning it, imputing missing values, and normalizing it. It then trains an artificial intelligence model using machine learning frameworks like TensorFlow and PyTorch.

[1097] Output: A trained AI model is generated.

[1098] Step 3:

[1099] Input: Call from scammer

[1100] How it works: The device uses VoIP technology to receive calls from scammers. If a call comes in from a scammer, the device automatically receives and records the call.

[1101] Output: Recorded audio data is obtained.

[1102] Step 4:

[1103] Input: Recorded audio data

[1104] How it works: The device uses Google Cloud Speech-to-Text to convert recorded audio data into text data in real time.

[1105] Output: The converted character data is generated and sent to the server.

[1106] Step 5:

[1107] Input: Character data sent from the terminal

[1108] How it works: The server uses NLTK and spacy to process the received text data and uses AI models to determine the likelihood of fraud. For example, a sentence like "Your bank account is at risk" would be considered a sign of fraud.

[1109] Output: The result is that there is a high possibility of fraud.

[1110] Step 6:

[1111] Input: Character data when it is determined that there is a high possibility of fraud

[1112] How it works: The server uses a generative AI model such as GPT-4 to generate natural-sounding elderly-sounding responses, such as "I'm sorry, what should I do?"

[1113] Output: The generated dialogue response text is generated and sent to the terminal.

[1114] Step 7:

[1115] Input: Dialogue response text sent from the server

[1116] How it works: The device uses a speech synthesis service such as Amazon Polly to convert dialogue response text into an elderly-sounding voice.

[1117] Output: The generated audio data is obtained and sent to the impostor.

[1118] Step 8:

[1119] Input: Scammer's voice response data

[1120] How it works: The scammer's responses are recorded again and converted to text using Google Cloud Speech-to-Text, which is then sent back to the server.

[1121] Output: The scammer's response text data is obtained.

[1122] Step 9:

[1123] Input: Scammer's response text data sent to the server

[1124] How it works: The server compiles the characteristics and identifying information of fraudsters and shares them in real time with relevant authorities (such as the police) using a "real-time data sharing API" or "RESTful API."

[1125] Output: Information about the fraudster that has been shared with the appropriate authorities.

[1126] (Application example 1)

[1127] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1128] Telephone scams by fraudsters are becoming more sophisticated every day, resulting in many victims. Elderly people are particularly vulnerable to fraud, making countermeasures urgently needed. However, current security systems have difficulty detecting fraud in real time and taking countermeasures. There is also the risk that victims may have to confront the fraudsters themselves. Therefore, there is a need to automate the entire process of receiving and identifying fraudulent phone calls, generating dialogue responses, collecting information, and reporting to the relevant authorities, in order to prevent damage before it occurs.

[1129] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1130] In this invention, the server includes: means for collecting data on fraud; means for training an artificial intelligence model to learn fraud patterns based on the collected data; means for receiving calls from fraudsters and converting the content of the calls into text; means for determining the possibility of fraud using the converted text; means for generating a dialogue response if there is a possibility of fraud; means for converting the generated dialogue response into voice and responding to the fraudster; means for collecting information on the fraudster and sharing the information with relevant institutions; means for the smart device to receive calls and convert the voice into text in real time; and means for issuing a warning to the user based on the determination result and generating an elderly-style response. This automates the process from receiving fraudulent calls to determining whether they are fraudulent, generating a dialogue response, collecting information, and reporting to relevant institutions, making it possible to protect victims, especially elderly people, from fraud.

[1131] "Fraud Data" means information relating to fraudulent activity, including data about the fraudster's methods, means of communication, details of the victim, and the location and time of the fraud.

[1132] An "artificial intelligence model" is an algorithm that learns patterns from large amounts of data and determines the likelihood of fraud based on new data.

[1133] "Means for converting to text" refers to the process of converting voice data into text data using voice recognition technology.

[1134] The "means for determining the likelihood of fraud" is an algorithm or machine learning model that analyzes the collected text data and determines whether it constitutes fraudulent activity.

[1135] The "means for generating a dialogue response" is a process that automatically creates an appropriate response to continue dialogue with the fraudster when it is determined that there is a high possibility of fraud.

[1136] The "means for converting to speech" is a speech synthesis technology for converting the generated text dialogue response into speech output.

[1137] "Means for sharing with relevant authorities" refers to a data transmission function that allows information about fraudsters to be reported to relevant authorities and security agencies.

[1138] "Means by which a smart device receives a call" refers to the process by which a smartphone or other mobile device receives a call from a scammer and initiates processing.

[1139] "Means for converting to text in real time" refers to speech recognition technology that instantly converts received speech into text data.

[1140] The "means of issuing a warning to the user and generating an elderly-sounding response" is a mechanism that, if it is determined that there is a high possibility of fraud, displays a fraud warning to the user and generates a response that allows the user to continue a natural dialogue with the fraudster.

[1141] The system of this invention provides a specific process for automatically detecting fraudulent calls and taking countermeasures. This system is mainly composed of three parts: a server, a terminal, and a user.

[1142] Server Processing

[1143] 1. Data Collection:

[1144] The server periodically collects fraud data from fraud databases and other sources on the internet, including information about fraudsters' tactics, methods of communication, victim details, and location and time of occurrence.

[1145] 2. Training the AI ​​model:

[1146] The server trains artificial intelligence models to identify and learn from the collected data, which are then used to update criteria and algorithms for determining potential fraud.

[1147] 3. Fraud determination:

[1148] The server analyzes the text data sent from the device during the incoming call, and uses that data to determine the possibility of fraud using an AI model.

[1149] 4. Response Generation:

[1150] If fraud is determined to be likely, the server generates natural responses in an elderly voice that serve to continue the dialogue with the fraudster and elicit further information.

[1151] 5. Data Sharing:

[1152] The characteristics and identifying information of fraudsters are collected, collated, and shared with the relevant authorities (police, etc.) in real time, which will contribute to the identification and arrest of fraudsters.

[1153] Terminal handling

[1154] 1. Receiving a call:

[1155] The device will always be available to receive calls using an unused phone number, and if a call comes in from a scammer, it will automatically receive and answer the call.

[1156] 2. Speech Recognition:

[1157] The conversation with the scammer is recorded in real time and converted into text using speech recognition technology. This converted text data is sent to a server, which allows the received voice to be instantly converted into text. The hardware used could be a smartphone or mobile device, and the software could be Google Speech-to-Text API or IBM Watson Speech to Text.

[1158] 3. Audio conversion and transmission:

[1159] The response text received from the server is converted into an elderly-sounding voice, which generates a natural response that is easy for fraudsters to recognize and sends it to them. The voice synthesis technology used is gTTS (Google Text-to-Speech).

[1160] User (system administrator) processing

[1161] 1. System Monitoring:

[1162] Users can check whether the system is operating normally through dashboards and monitoring tools that monitor the overall system status, and if an abnormality occurs, they can take prompt action.

[1163] 2. System optimization:

[1164] Users can adjust system parameters based on collected data and system performance data, including retraining AI models as new fraud techniques are discovered.

[1165] Specific examples

[1166] For example, consider the case where a fraudster calls and says, "Your bank account is at risk. We need your immediate attention!" First, the device automatically receives the call and converts the recorded voice into text in real time. This converted text is sent to a server, which analyzes the received text and determines that it is likely a fraud. Next, the device generates an elderly-sounding response such as "I'm sorry, what should I do?" and sends it to the device. The device then converts the received response text into voice and replies to the fraudster. The fraudster's response is recorded again and further information is sent to the server. This series of steps prevents fraud before it occurs.

[1167] Prompt Sentence Examples

[1168] This is an example of actual input to a generative AI model.

[1169] "Please recognize the voice of this call and determine if it is likely a scam. If it is, please display a warning and generate an elderly-sounding response to continue the conversation with the scammer."

[1170] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1171] Step 1:

[1172] Receiving a call

[1173] The terminal always keeps itself available to receive calls using an unused phone number. When a call from a scammer comes in, it automatically answers the call and starts recording. At this point, the input is the audio data of the received call, and the output is the recorded audio data.

[1174] Step 2:

[1175] Voice Recognition

[1176] The device converts the recorded voice data into text in real time using speech recognition technology. The input is the recorded voice data, and the output is text data. This process uses the Google Speech-to-Text API and IBM Watson Speech to Text.

[1177] Step 3:

[1178] Fraud judgment

[1179] The server receives the text data sent from the device and uses an artificial intelligence model to determine the likelihood of fraud. The input is the text data sent from the device, and the output is the determination result regarding the likelihood of fraud. A machine learning algorithm is used for data analysis.

[1180] Step 4:

[1181] Warning notice

[1182] If the server determines that there is a high possibility of fraud, it sends a warning to the terminal. The input is the result of the fraud determination, and the output is a warning to the user. This warning is displayed to the user through the terminal interface.

[1183] Step 5:

[1184] Response Generation

[1185] If the server determines that fraud is likely, it generates a natural response in the style of an elderly person. This generated response serves to continue the dialogue with the fraudster and elicit further information. The input is the result of the fraud detection, and the output is a response sentence in the style of an elderly person. A generative AI model is used to generate this response.

[1186] Step 6:

[1187] Voice conversion and transmission

[1188] The device converts the response text received from the server into an elderly-sounding voice. The voice synthesis technology used is gTTS (Google Text-to-Speech). The input is the generated response text, and the output is voice data. The converted voice is then sent back to the fraudster.

[1189] Step 7:

[1190] Scammers' information gathering and sharing

[1191] The device transmits the recorded voice data during the ongoing conversation with the fraudster to the server, which analyzes it, extracts the fraudster's characteristic information, and shares it with relevant organizations. The input is the recorded voice data, and the output is the analyzed characteristic information of the fraudster and the data to be shared with relevant organizations.

[1192] This series of steps will automate the entire process, from receiving fraudulent calls to identifying them, generating dialogue responses, collecting information, and reporting them to relevant authorities.

[1193] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1194] This invention is a system designed to receive and analyze calls from fraudsters and strengthen fraud prevention measures, and by combining it with an emotion engine to recognize user emotions, it further enhances accuracy. This system consists of three main components: a server, a terminal, and a user. The process and operation of each component are explained in detail below.

[1195] Server Processing

[1196] Data collection

[1197] The server periodically collects fraud data from online fraud databases and other sources. This data includes information about fraudsters' tactics, communication methods, victim details, location and time of occurrence. The collected data is stored in the server's internal database.

[1198] Training an AI model

[1199] Based on the collected data, the server trains an artificial intelligence model to learn fraud patterns, which includes criteria and algorithms to determine the likelihood of fraud.

[1200] Fraud judgment

[1201] The server analyzes the text data of the incoming call sent from the device and uses an AI model to determine the possibility of fraud. If it determines that there is a high possibility of fraud, it proceeds to the next step.

[1202] Response Generation

[1203] If fraud is determined to be likely, the server generates a natural-sounding response in the style of an elderly person, which is used to continue the dialogue with the fraudster and extract further information. The generated response text is then sent to the device.

[1204] Data Sharing

[1205] Collect and organize information about the characteristics and identity of fraudsters, and share it with the relevant authorities (police, etc.) in real time. This information sharing will contribute to the identification and arrest of fraudsters.

[1206] Terminal handling

[1207] Receiving a call

[1208] The device will use an unused phone number and will always be available to receive calls. If a scammer calls, the device will automatically receive and answer the call.

[1209] Voice Recognition

[1210] The device records the fraudster's voice in real time and converts it into text using voice recognition technology, which is then sent to a server.

[1211] Voice conversion and transmission

[1212] The response text received from the server is converted into an elderly-sounding voice, and the conversation continues by replying to the fraudster with the converted voice.

[1213] emotion recognition

[1214] The emotion engine installed on the device analyzes the user's emotions during the conversation with the scammer. For example, emotions such as anger, anxiety, and tension can be detected from the scammer's voice. This emotional information is sent to the server and reflected in the response generation.

[1215] User (system administrator) processing

[1216] System Monitoring

[1217] Users can check the dashboard to monitor the overall system status and quickly respond to any abnormalities or errors that occur.

[1218] System Optimization

[1219] Users can adjust system parameters based on collected data and system performance data, and retrain AI models to adapt to new fraudulent schemes.

[1220] Specific examples

[1221] As a specific scenario, let's consider a situation where a scammer calls and says, "Your bank account has been compromised. We need your immediate attention."

[1222] Terminal handling:

[1223] The device automatically receives the call and records the audio, which is then converted into text using voice recognition technology and sent to a server.

[1224] Server Action:

[1225] The server analyzes the received text, determines it is likely a scam, and then generates an elderly-sounding response to the device: "Sorry, what do I do?"

[1226] Terminal handling:

[1227] The device converts the received response text into speech and responds to the fraudster, while simultaneously analyzing the emotions in the fraudster's voice and sending the emotional information to the server.

[1228] Server Action:

[1229] The server uses this emotional information to generate an appropriate response that reflects the fraudster's emotional state when generating the next response. For example, if the fraudster is feeling anxious, the server can use this to devise a response that will further encourage the fraudulent tactics.

[1230] In this way, the server, terminal, and user work together to communicate with fraudsters in order to prevent fraud before it happens. Furthermore, by combining it with an emotion engine, the system can analyze the emotions of fraudsters, enabling more effective dialogue and information gathering.

[1231] The processing flow will be explained below.

[1232] Step 1:

[1233] Data collection (server processing)

[1234] The server periodically collects fraud data from fraud databases on the Internet and other sources. This data includes information about fraudsters' tactics, communication methods, victim details, location and time of occurrence. This collected data is stored in an internal database.

[1235] Step 2:

[1236] AI model training (server processing)

[1237] The server uses the collected data to retrain its artificial intelligence model, which learns fraud patterns, allowing it to adapt to the latest fraud techniques. The trained AI model is then used to identify fraud and generate responses for the next transaction.

[1238] Step 3:

[1239] Waiting for and receiving calls (terminal processing)

[1240] The device will use an unused phone number and will always be available to receive calls. If a scammer calls, the device will automatically answer the call and initiate the call.

[1241] Step 4:

[1242] Speech recognition and text conversion (device processing)

[1243] The device records the fraudster's voice in real time and converts this voice data into text using speech recognition technology, which is then sent to a server.

[1244] Step 5:

[1245] Fraud detection (server processing)

[1246] The server analyzes the text data sent from the device and uses an AI model to determine the likelihood of fraud. If it determines that there is a high possibility of fraud, it proceeds to the next step.

[1247] Step 6:

[1248] Emotion recognition (device processing)

[1249] An emotion engine on the device analyzes the scammer's voice and determines their emotional state (e.g., anger, impatience, calm, etc.) This emotion data is sent to the server for use in generating subsequent responses.

[1250] Step 7:

[1251] Response generation (server processing)

[1252] The server generates appropriate responses in the style of an elderly person based on the fraudster's tactics and emotional data. The responses are intended to continue the dialogue with the fraudster and elicit further information. The generated response text is then sent to the device.

[1253] Step 8:

[1254] Audio conversion and transmission (terminal processing)

[1255] The device uses speech synthesis technology to convert the response text received from the server into an elderly-sounding voice, which is then sent back to the fraudster, continuing the dialogue.

[1256] Step 9:

[1257] Information collection (server processing)

[1258] The server analyzes the information collected through interactions with the fraudsters and stores their characteristics (phone number, tactics, emotional state, etc.) in a comprehensive database.

[1259] Step 10:

[1260] Information sharing (server processing)

[1261] The server then shares the collected and analyzed information on fraudsters with the police and other relevant authorities in real time. This information sharing process facilitates the identification and arrest of fraudsters.

[1262] Step 11:

[1263] System monitoring (user actions)

[1264] Users can use a dashboard to monitor the overall system operation status and respond quickly to any errors or abnormalities that occur. Daily monitoring maintains the reliability and stability of the system.

[1265] Step 12:

[1266] System optimization (user processing)

[1267] Users can analyze the system's performance data and adjust configuration parameters as needed, including instructing the AI ​​model to retrain to adapt to new fraudulent schemes.

[1268] In a specific scenario, if a fraudster calls and says, "Your bank account is at risk. Immediate action is required," the system works as follows: The device receives the call and uses voice recognition technology to convert the content into text and send it to the server. The server analyzes the text, determines whether it is likely to be fraudulent, generates a response text, and sends it to the device. The device then converts the response text into speech and replies to the fraudster. During this process, the emotion engine analyzes the fraudster's emotions, and this data is reflected in the generation of the next response.

[1269] Example 2

[1270] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1271] In recent years, fraud has become more sophisticated and diverse, making it difficult to respond to conventional countermeasures. Telephone fraud targeting the elderly has been increasing in particular, and effective countermeasures are needed. Systems are needed that not only prevent fraud from occurring, but also collect and share information in real time to identify and arrest fraudsters. Furthermore, there is a need to expose fraudulent activities by analyzing fraudsters' emotions and engaging in more effective dialogue.

[1272] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting data related to fraud, means for training an artificial intelligence model to learn fraud patterns based on the collected data, means for receiving communications from fraudsters and converting the content of the communications into text, means for determining the possibility of fraud using the converted text, means for generating a dialogue response if there is a possibility of fraud, means for converting the generated dialogue response into voice and replying to the fraudster, means for analyzing the characteristics of the fraudster and obtaining emotional information, and means for sharing information about the fraudster with relevant agencies. This improves the accuracy of fraud countermeasures, prevents fraudulent acts, and enables the collection and sharing of information to identify and arrest fraudsters in real time.

[1273] "Fraud Data" means information relating to fraudulent activity, including information such as the fraudulent method, means of communication, details of the victim, and the location and time of the occurrence.

[1274] An "artificial intelligence model" refers to an algorithm that learns from data and automatically performs specific tasks.

[1275] "Communications" refers to the means of communication used by scammers, such as voice or messages.

[1276] "Means for converting to text" refers to speech recognition technology for converting voice or messages into text data.

[1277] "Means for determining likelihood of fraud" refers to algorithms or models that analyze the converted text data and assess the risk of fraud.

[1278] "Means for generating dialogue responses" refers to an algorithm or generative model for automatically generating appropriate responses to continue the dialogue with the fraudster.

[1279] "Means for converting to speech" refers to speech synthesis technology for playing the generated text dialogue response as speech.

[1280] "Means of obtaining emotional information" refers to technology that analyzes emotions from the fraudster's voice and messages and identifies emotional states such as anger or impatience.

[1281] "Means of sharing information with relevant agencies" refers to communication methods and interfaces for providing information about fraudsters to relevant agencies, such as the police, in real time.

[1282] This invention is a system designed to prevent fraud by receiving calls from fraudsters, analyzing them, and strengthening countermeasures. The system consists of three main components: a server, a terminal, and a user, which work together.

[1283] Server Processing

[1284] Data collection

[1285] The server collects fraud data from online fraud databases and other sources, including fraudster tactics, communication methods, victim details, location and time of occurrence. Specifically, it retrieves data from the ScamWatch and FraudLog databases using HTTP requests and stores it in the server's internal database.

[1286] Training an AI model

[1287] The collected data is preprocessed and an artificial intelligence model is trained. For example, TensorFlow or PyTorch is used to learn fraud patterns. Specific features such as specific methods and time periods are incorporated into the algorithm to improve the accuracy of fraud detection.

[1288] Analysis of incoming calls and fraud detection

[1289] The server receives the text data of the incoming call sent from the device and analyzes it using an AI model. The analysis uses keywords and phrases contained in the text data to determine the possibility of fraud. For example, phrases such as "bank account," "risk," and "action required" are determined to indicate a high risk of fraud.

[1290] Response Generation

[1291] If fraud is deemed likely, the server uses a generative AI model (e.g., OpenAI GPT-4) to generate natural-sounding responses in the style of an elderly person, designed to continue the dialogue with the fraudster and elicit further information, such as "I'm sorry, what should I do?"

[1292] Data Sharing

[1293] The characteristics and identifying information of fraudsters are collated and shared with the relevant authorities (e.g., police) in real time. This information is sent via API to expedite response.

[1294] Terminal handling

[1295] Receiving a call

[1296] The device is set to an unused phone number and is always ready to receive calls. When a call comes in from a scammer, it automatically receives the call and prepares to record it. Specifically, it uses Twilio's call receiving API.

[1297] Speech recognition and text conversion

[1298] The device records the fraudster's voice in real time and converts it into text data using Google Cloud Speech-to-Text. The recorded audio stream is sent to an API, which outputs the text data, which is then sent to the server.

[1299] Voice conversion and transmission

[1300] The response text received from the server is converted into an elderly-sounding voice. Specifically, speech synthesis is performed using Amazon Polly or Google Text-to-Speech. The converted voice is then sent back to the fraudster, and the dialogue continues.

[1301] emotion recognition

[1302] The device uses the Microsoft Azure Emotion API to analyze emotional information from the fraudster's voice, detecting emotions such as anger, worry, and tension, and sending this emotional information to a server.

[1303] User (system administrator) processing

[1304] System Monitoring

[1305] Users use Grafana and Kibana to monitor the overall system status. If an abnormality or error occurs, they receive notifications and can respond quickly. They can understand the situation by checking log data and real-time notifications.

[1306] System Optimization

[1307] Users can adjust system parameters based on collected data and system performance data, and retrain the AI ​​model as needed to update the system to adapt to new fraudulent schemes.

[1308] Specific examples

[1309] As a specific scenario, let's consider a situation where a scammer calls and says, "Your bank account has been compromised. We need your immediate attention."

[1310] Terminal handling

[1311] The device automatically receives the call using Twilio, then converts the fraudster's speech into text in real time using Google Cloud Speech-to-Text, which then sends the text data to the server via an RTP stream.

[1312] Server Processing

[1313] The server receives the text data, "Your bank account is at risk. Immediate action is required." and analyzes it using an AI model. If it determines there is a high possibility of fraud, it uses OpenAI's GPT-4 to generate an elderly-sounding response, "I'm sorry, what should I do?", and sends it to the device.

[1314] Terminal handling

[1315] The device uses Amazon Polly to convert the received response text into an elderly-sounding voice and responds to the fraudster. At the same time, it uses Microsoft Azure's Emotion API to analyze emotional information (such as impatience or tension) from the fraudster's voice and sends it to the server.

[1316] Server Processing

[1317] The server will use this emotional information to generate the next response. For example, if it determines that the fraudster is anxious, it can generate a question like, "I'd like to confirm again, is your account really at risk?" This can further refine the fraudulent scheme.

[1318] Example prompts to input to the generative AI model

[1319] "Please translate the following sentence into a natural response from a senior citizen: 'Your bank account is at risk. Action is required immediately.'"

[1320] This prompt can be used to automatically generate an appropriate response.

[1321] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1322] Server Processing

[1323] Step 1: Data collection

[1324] The server collects fraud data from fraud databases and other sources on the internet. Specifically, it uses HTTP requests to retrieve information from the database about the fraud scheme, communication method, victim details, location and time of the incident. The input is the database API endpoint, and the output is detailed data about the fraud. The retrieved data is stored in the server's internal database.

[1325] Step 2: Training the AI ​​model

[1326] The server preprocesses the collected data and trains an artificial intelligence model. Specifically, it uses TensorFlow and PyTorch to learn fraud patterns. The input is the preprocessed dataset, and the output is a trained AI model. Features such as specific methods and time periods are reflected in the algorithm to improve the model's accuracy.

[1327] Step 3: Analyze incoming calls and identify fraud

[1328] The server receives the text data of the incoming call sent from the device and analyzes it using an AI model. The input is text data, and the output is a judgment result indicating the possibility of fraud. In this process, keywords and phrases contained in the text are analyzed to assess the risk of fraud.

[1329] Step 4: Response Generation

[1330] If the server determines that there is a high possibility of fraud, it uses a generative AI model to generate a natural-sounding response in the style of an elderly person. The input is the judgment result and the fraudster's text data, and the output is the generated response text. For example, it generates a response such as, "I'm sorry, what should I do?"

[1331] Step 5: Data sharing

[1332] The server compiles the characteristics and identifying information of fraudsters and shares it with the relevant authorities in real time. The input is detailed information about the fraudsters, and the output is the information sent to the relevant authorities. The information is sent quickly through the API, contributing to the identification and arrest of fraudsters.

[1333] Terminal handling

[1334] Step 1: Receiving a call

[1335] The terminal is set to an unused phone number and is always ready to receive calls. The input is a call from a fraudster, and the output is the received voice data. Specifically, it uses Twilio's call receiving API to automatically receive calls.

[1336] Step 2: Speech recognition and text conversion

[1337] The device records the fraudster's voice in real time and converts it into text data using Google Cloud Speech-to-Text. The input is audio data and the output is text data. The recorded audio stream is sent to the API, and the converted text is sent to the server.

[1338] Step 3: Convert and send

[1339] The device converts the response text received from the server into an elderly-sounding voice. The input is the response text, and the output is synthesized voice data. Specifically, Amazon Polly and Google Text-to-Speech are used to generate the voice and respond to the scammer.

[1340] Step 4: Emotion Recognition

[1341] An emotion engine implemented on the device is used to analyze emotions from the fraudster's voice. The input is the fraudster's voice data, and the output is emotional information (e.g., anger, tension, etc.). The emotional information is sent to the server and reflected in the generation of the next response.

[1342] User (system administrator) processing

[1343] Step 1: System monitoring

[1344] Users check a dashboard to monitor the overall system operation status. The input is real-time system log data, and the output is the monitoring results. Specifically, Grafana and Kibana are used to monitor and receive notifications for abnormalities and errors.

[1345] Step 2: System optimization

[1346] Users adjust the system parameters based on collected data and system performance data. The input is performance data and information on new fraud schemes, and the output is optimized system settings. If necessary, the AI ​​model can also be retrained.

[1347] (Application example 2)

[1348] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1349] The number of fraudulent phone calls, often involving sophisticated tactics by scammers, is steadily increasing, with elderly people being particularly vulnerable to these attacks. Current systems for combating fraudulent phone calls often struggle to identify these calls, resulting in delayed effective responses. Furthermore, they lack the means to analyze the emotions and psychological state of fraudsters, making it difficult to effectively deter and collect information from them. This can result in investigative authorities being unable to respond quickly, potentially exacerbating the damage. Therefore, there is a need for a system that can improve the accuracy and effectiveness of countermeasures against fraudulent phone calls by recognizing the emotions of fraudsters and generating responses that sound like an elderly person.

[1350] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1351] In this invention, the server includes means for collecting data on fraud, means for training an artificial intelligence model that learns fraud patterns based on the collected data, means for determining the possibility of fraud using the converted text, means for generating an elderly-sounding dialogue response if there is a possibility of fraud, means for converting the generated dialogue response into voice and replying to the fraudster, means for analyzing emotions from the fraudster's voice, and means for collecting information on the fraudster and sharing it with relevant institutions. This makes it possible to provide an appropriate response according to the fraudster's emotional state, effectively continue the dialogue to collect information, and ultimately prevent fraud before it occurs.

[1352] "Fraud Data" means any information relating to fraudulent activity, including the method of fraud, means of communication, details of the victim, location and time of the occurrence.

[1353] An "artificial intelligence model" refers to a computer model that can learn from large amounts of data and identify and analyze specific patterns and features.

[1354] "Convert to text" refers to the process of converting audio data into text data.

[1355] "Means for determining the likelihood of fraud" refers to the process of using artificial intelligence models or other algorithms to analyze and determine whether fraud is suspected.

[1356] "Elderly-style dialogue responses" refers to text and voice responses generated to make it appear as if an elderly person is having a conversation.

[1357] "Convert to voice" refers to the process of converting text data into voice data.

[1358] "Means for analyzing emotions from fraudsters' voices" refers to technology for identifying the emotional state (e.g., anger, impatience, tension, etc.) of fraudsters from their vocalizations.

[1359] "Fraudster information" refers to any information that can be used to identify a fraudster, including voice characteristics, phone number, words used, and conversation content.

[1360] "Means of sharing with relevant agencies" refers to the process for providing collected and analyzed fraud information to investigative agencies such as the police and monitoring agencies.

[1361] The present invention is a system for dealing with fraudulent calls from fraudsters, and is composed of a server, a terminal, and a user. The processing of the entire system will be described in detail below.

[1362] Server Processing

[1363] Data collection

[1364] The server periodically collects fraud data from online fraud databases and other sources. This data includes information about the fraud scheme, means of communication, victim details, location and time of occurrence. The collected data is stored in the server's internal database.

[1365] Training an AI model

[1366] Based on the collected data, the server trains an artificial intelligence model to learn fraud patterns, specifically by using pattern recognition techniques and machine learning algorithms to extract and learn the characteristics of suspicious communications.

[1367] Fraud judgment

[1368] The server analyzes the text data of the incoming call sent from the terminal and uses an artificial intelligence model to determine the possibility of fraud. If the result of this determination indicates a high possibility of fraud, the system proceeds to the next step.

[1369] Response Generation

[1370] If the server determines that fraud is likely, it generates a natural response in the style of an elderly person. This response is used to continue the dialogue with the fraudster and extract further information. The generated response text is sent to the device.

[1371] Data Sharing

[1372] Collecting characteristics and identifying information of fraudsters and sharing it with relevant authorities (such as the police) in real time, which will contribute to identifying and arresting fraudsters.

[1373] Terminal handling

[1374] Receiving a call

[1375] The device will always be ready to receive calls, and if a scammer calls you, it will automatically receive and answer the call.

[1376] Voice Recognition

[1377] The device records the fraudster's voice in real time and converts it into text using voice recognition technology (e.g., Google Cloud Speech-to-Text), which is then sent to a server.

[1378] Voice conversion and transmission

[1379] The response text received from the server is converted into an elderly-sounding voice. The converted voice is then sent back to the fraudster, continuing the dialogue. Tools such as Google Text-to-Speech (gTTS) are used for voice synthesis.

[1380] emotion recognition

[1381] An emotion engine (e.g., IBM Watson Tone Analyzer) installed on the device analyzes the emotions of the fraudster during the conversation. The analysis results are sent to the server and reflected in the response generation.

[1382] User (system administrator) processing

[1383] System Monitoring

[1384] Users can check the dashboard to monitor the overall system status, and respond quickly if an abnormality or error occurs.

[1385] System Optimization

[1386] Users can adjust system parameters based on collected data and system performance data, and retrain AI models to adapt to new fraudulent schemes.

[1387] Specific examples

[1388] For example, imagine a scammer calling you and saying, "Your bank account is at risk. We need your immediate attention."

[1389] 1. Terminal Processing

[1390] The device automatically receives the call and records the audio, which is then converted into text using voice recognition technology and sent to a server.

[1391] 2. Server Processing

[1392] The server analyzes the received text and determines that it is likely a scam, then generates an elderly-sounding response to the user, such as "Really? What should I do?", and sends it to the device.

[1393] 3. Terminal Processing

[1394] The device converts the received response text into speech and responds to the fraudster. It also analyzes the emotions in the fraudster's voice and sends the emotional information to the server.

[1395] 4. Server Processing

[1396] The server then uses the emotional information to generate an appropriate response that reflects the fraudster's emotional state. For example, if the fraudster is feeling anxious, the server can use this to devise a response that will further encourage the fraudulent tactics.

[1397] Example prompt sentence:

[1398] "Pretend to be a senior citizen and respond naturally to the following statement: 'Your bank account is at risk.'"

[1399] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1400] Step 1:

[1401] Receiving a call (terminal)

[1402] The device is always ready to receive calls. When a call comes in from a fraudster, it automatically answers the call and starts the conversation. The input is the call signal from the fraudster, and the output is the call audio data.

[1403] Step 2:

[1404] Voice recording and recognition (device)

[1405] The device records the scammer's voice in real time and converts the voice data into text data using voice recognition technology such as Google Cloud Speech-to-Text. The input is the recorded voice data, and the output is text data.

[1406] Step 3:

[1407] Sending text data (terminal → server)

[1408] The terminal sends the converted text data to the server. The input is the text data generated by the speech recognition technology, and the output is the text data sent to the server.

[1409] Step 4:

[1410] Fraud detection (server)

[1411] The server analyzes the received text data and uses an artificial intelligence model (generative AI model) to determine the likelihood of fraud. The input is text data, and the output is a determination result indicating the likelihood of fraud. This data is processed through data calculations and analysis within the server.

[1412] Step 5:

[1413] Generate response (server)

[1414] If the server determines that there is a high possibility of fraud, it generates a natural response in the style of an elderly person for the next dialogue step, where it uses a generative AI model and a prompt to generate an appropriate response. The input is the fraud detection result and the prompt, and the output is the generated response text.

[1415] Step 6:

[1416] Sending a response (server → terminal)

[1417] The server sends the generated response text to the terminal. The input is the generated response text, and the output is the text data sent to the terminal.

[1418] Step 7:

[1419] Voice conversion and reply (terminal)

[1420] The device converts the received response text into an elderly-sounding voice and generates an audio file using Google Text-to-Speech (gTTS), which then replies to the fraudster using this generated voice. The input is the response text, and the output is the voice response to the fraudster.

[1421] Step 8:

[1422] Emotion Recognition (Device)

[1423] The device analyzes the emotions from the fraudster's voice and sends the results to a server, using technologies such as IBM Watson Tone Analyzer. The input is the fraudster's voice data, and the output is the analyzed emotion data.

[1424] Step 9:

[1425] Receiving and responding to emotion data (server)

[1426] The server receives the emotion data sent from the device and reflects it when generating the next response based on that emotion information. This makes it possible to generate an appropriate response based on the fraudster's psychological state. The input is emotion data, and the output is the next response text based on the emotion.

[1427] Step 10:

[1428] Fraudster information collection and sharing (server)

[1429] The server organizes and collects information obtained from fraudsters and shares it with the relevant authorities (police, etc.) in real time. The input is the fraudster information obtained from the conversation, and the output is the shared information sent to the relevant authorities.

[1430] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1431] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1432] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1433] [Fourth embodiment]

[1434] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1435] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1436] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1437] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1438] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1439] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1440] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1441] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1442] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1443] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1444] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1445] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1446] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1447] The system of this invention is designed to prevent and expose fraudulent activities by receiving calls from fraudsters, analyzing the content of the conversation, and reporting it to the relevant authorities. Below, we will explain the specific processing and operation of this system, dividing it into three roles: server, terminal, and user.

[1448] Server Processing

[1449] Data collection:

[1450] The server periodically collects fraud data from fraud databases and other sources on the internet, including information about fraudster tactics, methods of communication, victim details, and the location and time of fraud.

[1451] Training the AI ​​model:

[1452] The server uses the collected data to train artificial intelligence models to identify and learn from fraud patterns, which are then used to update criteria and algorithms for determining potential fraud.

[1453] Fraud judgment:

[1454] The server analyzes the text data sent from the device and uses that data to determine the possibility of fraud using an AI model.

[1455] Response generation:

[1456] If fraud is determined to be likely, the server generates natural responses in an elderly voice that serve to continue the dialogue with the fraudster and elicit further information.

[1457] Data sharing:

[1458] The characteristics and identifying information of fraudsters are collected, collated, and shared with the relevant authorities (police, etc.) in real time, which will contribute to the identification and arrest of fraudsters.

[1459] Terminal handling

[1460] Receiving a call:

[1461] The device will always be available to receive calls using an unused phone number, and will automatically receive and answer calls from scammers.

[1462] Voice Recognition:

[1463] The conversation with the scammer is recorded in real time and converted into text using voice recognition technology, which is then sent to a server.

[1464] Audio conversion and transmission:

[1465] The response text received from the server is converted into an elderly-sounding voice, which generates a natural response that is easy for fraudsters to recognize and sends it to the fraudster.

[1466] User (system administrator) processing

[1467] System Monitoring:

[1468] Users can check whether the system is operating normally through dashboards and monitoring tools that monitor the overall system status, and if an abnormality occurs, they can take prompt action.

[1469] System optimization:

[1470] Based on the collected data and system performance data, users can adjust system parameters as needed, including retraining AI models as new fraud techniques are discovered.

[1471] Specific examples

[1472] To give a concrete scenario, consider a situation where a scammer calls and says, "Your bank account is at risk. We need your immediate attention!"

[1473] Terminal handling:

[1474] The device automatically receives the call and converts the recorded voice into text in real time, which is then sent to the server.

[1475] Server Action:

[1476] The server analyzes the received text, determines it is likely a scam, and then generates an elderly-sounding response to the device: "Sorry, what do I do?"

[1477] Terminal handling:

[1478] The device converts the received response text into speech and responds to the fraudster, then records the fraudster's response again and sends it to a server to collect the fraudster's information.

[1479] In this way, the server, terminal, and user work together to interact with fraudsters, and collect, analyze, and share the information necessary to identify them, thereby realizing a system that prevents fraudulent acts before they occur.

[1480] The processing flow will be explained below.

[1481] Step 1:

[1482] Data collection (server processing)

[1483] The server periodically collects fraud data from internet fraud databases and other sources. This data includes information about fraudsters' tactics, communication methods, victim details, location and time of occurrence. This data is stored in an internal database.

[1484] Step 2:

[1485] AI model training (server processing)

[1486] The server uses the collected data to retrain the artificial intelligence model to learn fraud patterns, which is then used to identify fraud and generate responses the next time.

[1487] Step 3:

[1488] Waiting for and receiving calls (terminal processing)

[1489] The device will always be available to receive calls using an unused phone number. If a scammer calls, the device will automatically answer the call and start the conversation.

[1490] Step 4:

[1491] Speech recognition and text conversion (device processing)

[1492] The device records the fraudster's voice in real time and converts it into text using speech recognition technology, which is then sent to a server.

[1493] Step 5:

[1494] Fraud detection (server processing)

[1495] The server analyzes the text data sent from the device and uses an AI model to determine whether there is a high possibility of fraud. If it is determined that there is a high possibility of fraud, it proceeds to the next step.

[1496] Step 6:

[1497] Response generation (server processing)

[1498] The server generates an appropriate elderly-style response based on the fraudulent scheme, which is used to continue the dialogue with the fraudster. The generated response text is sent to the device.

[1499] Step 7:

[1500] Audio conversion and transmission (terminal processing)

[1501] The device receives the response text from the server and converts it into an elderly-sounding voice using speech synthesis technology. The conversation continues when the device responds to the fraudster with the converted voice.

[1502] Step 8:

[1503] Information collection (server processing)

[1504] The server continuously analyzes the information obtained from interactions with the fraudsters and collects their characteristics and specific information (phone number, methods, etc.) in a database.

[1505] Step 9:

[1506] Information sharing (server processing)

[1507] The server collects and analyzes information on fraudsters and shares it with relevant agencies such as the police in real time, which contributes to the identification and arrest of fraudsters.

[1508] Step 10:

[1509] System monitoring (user actions)

[1510] Users regularly check the dashboard to monitor the system's operating status, and if an abnormality or error occurs, they can quickly take action.

[1511] Step 11:

[1512] System optimization (user processing)

[1513] Users analyze system performance data and adjust configuration parameters as needed, as well as retrain AI models to adapt to new fraudulent schemes.

[1514] Example 1

[1515] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1516] Fraudulent phone calls have become a social problem, with the methods becoming more sophisticated and the damage caused by them increasing. Fraud targeting the elderly in particular often leads to serious consequences. Furthermore, fraudsters often use international networks, making it difficult to deal with them using conventional methods. For this reason, there is a need for early detection and response to fraudulent phone calls, but current systems have difficulty determining fraud in real time, gathering information through dialogue, and identifying fraudsters. This article presents a system to solve this problem.

[1517] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1518] In this invention, the server includes means for collecting data on fraud, means for training an artificial intelligence model that learns fraud patterns based on the collected data, means for receiving calls from fraudsters and converting the content of the calls into text data, means for determining the possibility of fraud using the converted text data, means for generating a dialogue response using natural language generation technology if there is a possibility of fraud, means for converting the generated dialogue response into voice and replying to the fraudster, and means for organizing the characteristics and information of the fraudster and sharing that information with the relevant authorities in real time. This enables immediate identification of fraudulent calls, information collection through real-time dialogue, and rapid information sharing with the relevant authorities.

[1519] "Fraud Data" refers to information relating to fraudulent activity, such as the modus operandi, means of communication, victim details, location and time of occurrence.

[1520] "Artificial intelligence model" refers to a program that uses machine learning algorithms to learn patterns of fraudulent activity and determine the likelihood of fraud.

[1521] "Character data" refers to data converted from audio into text format.

[1522] "Natural language generation technology" refers to technology that uses AI to generate natural human conversational sentences.

[1523] "Dialogue Response" refers to text or audio generated to respond in a conversation with the fraudster.

[1524] "Relevant Authorities" refers to the police and other legal authorities that monitor and investigate fraudulent activities.

[1525] "Real-time sharing" refers to immediately transmitting information about fraudulent activity to relevant authorities.

[1526] MODE FOR CARRYING OUT THE INVENTION

[1527] The system of this invention is a system that receives fraudulent phone calls, analyzes the content of the calls, and reports them to the relevant authorities to prevent fraud. Below, we will explain in detail the roles and specific processes of the server, terminal, and user.

[1528] Server Processing

[1529] Data collection:

[1530] The server periodically collects data from fraud databases and other sources on the internet, including information on the modus operandi, means of communication, victim details, location and time of occurrence, using information providers such as "Fraud Risk Database" and "ScanWatch."

[1531] Training the AI ​​model:

[1532] The server uses the collected data to train artificial intelligence models using machine learning frameworks such as TensorFlow and PyTorch, which are then used to build algorithms that learn fraud patterns and determine likelihood of fraud.

[1533] Fraud judgment:

[1534] The server receives the text data of the incoming call sent from the device. This data is analyzed using natural language processing tools such as NLTK and spacy. The AI ​​model determines the likelihood of fraud and, if there are signs of fraud, proceeds with further processing.

[1535] Response generation:

[1536] If fraud is deemed likely, the server uses a generative AI model such as GPT-4 to generate a natural-sounding elderly-sounding response that continues the dialogue with the fraudster and elicits further information.

[1537] Data sharing:

[1538] The server collects the characteristics and identifying information of fraudsters and shares it with the relevant authorities (police, etc.) in real time. This process uses a "real-time data sharing API" and a "RESTful API" to enable rapid data transmission.

[1539] Terminal handling

[1540] Receiving a call:

[1541] The device uses VoIP technology to keep unused phone numbers constantly available for calls, and automatically receives and records calls from scammers.

[1542] Voice Recognition:

[1543] The recorded voice is converted into text data in real time using Google Cloud Speech-to-Text technology, which is then sent to a server.

[1544] Audio conversion and transmission:

[1545] The response text received from the server is converted into an elderly-sounding voice using a speech synthesis service such as Amazon Polly, which is then sent back to the scammer to continue the conversation.

[1546] User (system administrator) processing

[1547] System Monitoring:

[1548] Users monitor the overall operation of the system through dashboards and monitoring tools such as Grafana and Nagios, and if an abnormality occurs in the system, users can take prompt action.

[1549] System optimization:

[1550] Users can adjust system parameters based on collected data and system performance data, and retrain the AI ​​model using Jupyter Notebook or Google Colab as new fraud schemes are discovered.

[1551] Specific examples

[1552] To give a concrete scenario, consider a situation where a scammer calls and says, "Your bank account is at risk. We need your immediate attention!"

[1553] 1. Your device will automatically receive this call and record the audio.

[1554] 2. The recorded audio is converted into text in real time using Google Cloud Speech-to-Text and sent to the server.

[1555] 3. The server analyzes the received text and uses an AI model to determine whether it is likely to be fraudulent.

[1556] 4. Next, generate an elderly-sounding response, "Sorry, what can I do?", and send it to the device.

[1557] 5. The device converts the received text into speech using Amazon Polly and responds to the scammer.

[1558] 6. The scammer's responses are recorded again and sent to the server to collect the scammer's information.

[1559] This process allows for the collection of necessary information while maintaining a dialogue with the scammer, enabling prompt reporting to the appropriate authorities. An example prompt is, "A scammer has called an elderly person to inform them of a risk to their bank account. Please generate a natural response that sounds like an elderly person."

[1560] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1561] Step 1:

[1562] Input: The server collects fraud data from fraud databases and other sources on the Internet.

[1563] How it works: The server periodically retrieves data using designated APIs and web scraping techniques, specifically using a "fraud risk database" and "information collection tools."

[1564] Output: The collected data is stored in a database on the server.

[1565] Step 2:

[1566] Input: Collected fraud data

[1567] How it works: The server preprocesses the collected data, which includes cleaning it, imputing missing values, and normalizing it. It then trains an artificial intelligence model using machine learning frameworks like TensorFlow and PyTorch.

[1568] Output: A trained AI model is generated.

[1569] Step 3:

[1570] Input: Call from scammer

[1571] How it works: The device uses VoIP technology to receive calls from scammers. If a call comes in from a scammer, the device automatically receives and records the call.

[1572] Output: Recorded audio data is obtained.

[1573] Step 4:

[1574] Input: Recorded audio data

[1575] How it works: The device uses Google Cloud Speech-to-Text to convert recorded audio data into text data in real time.

[1576] Output: The converted character data is generated and sent to the server.

[1577] Step 5:

[1578] Input: Character data sent from the terminal

[1579] How it works: The server uses NLTK and spacy to process the received text data and uses AI models to determine the likelihood of fraud. For example, a sentence like "Your bank account is at risk" would be considered a sign of fraud.

[1580] Output: The result is that there is a high possibility of fraud.

[1581] Step 6:

[1582] Input: Character data when it is determined that there is a high possibility of fraud

[1583] How it works: The server uses a generative AI model such as GPT-4 to generate natural-sounding elderly-sounding responses, such as "I'm sorry, what should I do?"

[1584] Output: The generated dialogue response text is generated and sent to the terminal.

[1585] Step 7:

[1586] Input: Dialogue response text sent from the server

[1587] How it works: The device uses a speech synthesis service such as Amazon Polly to convert dialogue response text into an elderly-sounding voice.

[1588] Output: The generated audio data is obtained and sent to the impostor.

[1589] Step 8:

[1590] Input: Scammer's voice response data

[1591] How it works: The scammer's responses are recorded again and converted to text using Google Cloud Speech-to-Text, which is then sent back to the server.

[1592] Output: The scammer's response text data is obtained.

[1593] Step 9:

[1594] Input: Scammer's response text data sent to the server

[1595] How it works: The server compiles the characteristics and identifying information of fraudsters and shares them in real time with relevant authorities (such as the police) using a "real-time data sharing API" or "RESTful API."

[1596] Output: Information about the fraudster that has been shared with the appropriate authorities.

[1597] (Application example 1)

[1598] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1599] Telephone scams by fraudsters are becoming more sophisticated every day, resulting in many victims. Elderly people are particularly vulnerable to fraud, making countermeasures urgently needed. However, current security systems have difficulty detecting fraud in real time and taking countermeasures. There is also the risk that victims may have to confront the fraudsters themselves. Therefore, there is a need to automate the entire process of receiving and identifying fraudulent phone calls, generating dialogue responses, collecting information, and reporting to the relevant authorities, in order to prevent damage before it occurs.

[1600] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1601] In this invention, the server includes: means for collecting data on fraud; means for training an artificial intelligence model to learn fraud patterns based on the collected data; means for receiving calls from fraudsters and converting the content of the calls into text; means for determining the possibility of fraud using the converted text; means for generating a dialogue response if there is a possibility of fraud; means for converting the generated dialogue response into voice and responding to the fraudster; means for collecting information on the fraudster and sharing the information with relevant institutions; means for the smart device to receive calls and convert the voice into text in real time; and means for issuing a warning to the user based on the determination result and generating an elderly-style response. This automates the process from receiving fraudulent calls to determining whether they are fraudulent, generating a dialogue response, collecting information, and reporting to relevant institutions, making it possible to protect victims, especially elderly people, from fraud.

[1602] "Fraud Data" means information relating to fraudulent activity, including data about the fraudster's methods, means of communication, details of the victim, and the location and time of the fraud.

[1603] An "artificial intelligence model" is an algorithm that learns patterns from large amounts of data and determines the likelihood of fraud based on new data.

[1604] "Means for converting to text" refers to the process of converting voice data into text data using voice recognition technology.

[1605] The "means for determining the likelihood of fraud" is an algorithm or machine learning model that analyzes the collected text data and determines whether it constitutes fraudulent activity.

[1606] The "means for generating a dialogue response" is a process that automatically creates an appropriate response to continue dialogue with the fraudster when it is determined that there is a high possibility of fraud.

[1607] The "means for converting to speech" is a speech synthesis technology for converting the generated text dialogue response into speech output.

[1608] "Means for sharing with relevant authorities" refers to a data transmission function that allows information about fraudsters to be reported to relevant authorities and security agencies.

[1609] "Means by which a smart device receives a call" refers to the process by which a smartphone or other mobile device receives a call from a scammer and initiates processing.

[1610] "Means for converting to text in real time" refers to speech recognition technology that instantly converts received speech into text data.

[1611] The "means of issuing a warning to the user and generating an elderly-sounding response" is a mechanism that, if it is determined that there is a high possibility of fraud, displays a fraud warning to the user and generates a response that allows the user to continue a natural dialogue with the fraudster.

[1612] The system of this invention provides a specific process for automatically detecting fraudulent calls and taking countermeasures. This system is mainly composed of three parts: a server, a terminal, and a user.

[1613] Server Processing

[1614] 1. Data Collection:

[1615] The server periodically collects fraud data from fraud databases and other sources on the internet, including information about fraudsters' tactics, methods of communication, victim details, and location and time of occurrence.

[1616] 2. Training the AI ​​model:

[1617] The server trains artificial intelligence models to identify and learn from the collected data, which are then used to update criteria and algorithms for determining potential fraud.

[1618] 3. Fraud determination:

[1619] The server analyzes the text data sent from the device during the incoming call, and uses that data to determine the possibility of fraud using an AI model.

[1620] 4. Response Generation:

[1621] If fraud is determined to be likely, the server generates natural responses in an elderly voice that serve to continue the dialogue with the fraudster and elicit further information.

[1622] 5. Data Sharing:

[1623] The characteristics and identifying information of fraudsters are collected, collated, and shared with the relevant authorities (police, etc.) in real time, which will contribute to the identification and arrest of fraudsters.

[1624] Terminal handling

[1625] 1. Receiving a call:

[1626] The device will always be available to receive calls using an unused phone number, and if a call comes in from a scammer, it will automatically receive and answer the call.

[1627] 2. Speech Recognition:

[1628] The conversation with the scammer is recorded in real time and converted into text using speech recognition technology. This converted text data is sent to a server, which allows the received voice to be instantly converted into text. The hardware used could be a smartphone or mobile device, and the software could be Google Speech-to-Text API or IBM Watson Speech to Text.

[1629] 3. Audio conversion and transmission:

[1630] The response text received from the server is converted into an elderly-sounding voice, which generates a natural response that is easy for fraudsters to recognize and sends it to them. The voice synthesis technology used is gTTS (Google Text-to-Speech).

[1631] User (system administrator) processing

[1632] 1. System Monitoring:

[1633] Users can check whether the system is operating normally through dashboards and monitoring tools that monitor the overall system status, and if an abnormality occurs, they can take prompt action.

[1634] 2. System optimization:

[1635] Users can adjust system parameters based on collected data and system performance data, including retraining AI models as new fraud techniques are discovered.

[1636] Specific examples

[1637] For example, consider the case where a fraudster calls and says, "Your bank account is at risk. We need your immediate attention!" First, the device automatically receives the call and converts the recorded voice into text in real time. This converted text is sent to a server, which analyzes the received text and determines that it is likely a fraud. Next, the device generates an elderly-sounding response such as "I'm sorry, what should I do?" and sends it to the device. The device then converts the received response text into voice and replies to the fraudster. The fraudster's response is recorded again and further information is sent to the server. This series of steps prevents fraud before it occurs.

[1638] Prompt Sentence Examples

[1639] This is an example of actual input to a generative AI model.

[1640] "Please recognize the voice of this call and determine if it is likely a scam. If it is, please display a warning and generate an elderly-sounding response to continue the conversation with the scammer."

[1641] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1642] Step 1:

[1643] Receiving a call

[1644] The terminal always keeps itself available to receive calls using an unused phone number. When a call from a scammer comes in, it automatically answers the call and starts recording. At this point, the input is the audio data of the received call, and the output is the recorded audio data.

[1645] Step 2:

[1646] Voice Recognition

[1647] The device converts the recorded voice data into text in real time using speech recognition technology. The input is the recorded voice data, and the output is text data. This process uses the Google Speech-to-Text API and IBM Watson Speech to Text.

[1648] Step 3:

[1649] Fraud judgment

[1650] The server receives the text data sent from the device and uses an artificial intelligence model to determine the likelihood of fraud. The input is the text data sent from the device, and the output is the determination result regarding the likelihood of fraud. A machine learning algorithm is used for data analysis.

[1651] Step 4:

[1652] Warning notice

[1653] If the server determines that there is a high possibility of fraud, it sends a warning to the terminal. The input is the result of the fraud determination, and the output is a warning to the user. This warning is displayed to the user through the terminal interface.

[1654] Step 5:

[1655] Response Generation

[1656] If the server determines that fraud is likely, it generates a natural response in the style of an elderly person. This generated response serves to continue the dialogue with the fraudster and elicit further information. The input is the result of the fraud detection, and the output is a response sentence in the style of an elderly person. A generative AI model is used to generate this response.

[1657] Step 6:

[1658] Voice conversion and transmission

[1659] The device converts the response text received from the server into an elderly-sounding voice. The voice synthesis technology used is gTTS (Google Text-to-Speech). The input is the generated response text, and the output is voice data. The converted voice is then sent back to the fraudster.

[1660] Step 7:

[1661] Scammers' information gathering and sharing

[1662] The device transmits the recorded voice data during the ongoing conversation with the fraudster to the server, which analyzes it, extracts the fraudster's characteristic information, and shares it with relevant organizations. The input is the recorded voice data, and the output is the analyzed characteristic information of the fraudster and the data to be shared with relevant organizations.

[1663] This series of steps will automate the entire process, from receiving fraudulent calls to identifying them, generating dialogue responses, collecting information, and reporting them to relevant authorities.

[1664] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1665] This invention is a system designed to receive and analyze calls from fraudsters and strengthen fraud prevention measures, and by combining it with an emotion engine to recognize user emotions, it further enhances accuracy. This system consists of three main components: a server, a terminal, and a user. The process and operation of each component are explained in detail below.

[1666] Server Processing

[1667] Data collection

[1668] The server periodically collects fraud data from online fraud databases and other sources. This data includes information about fraudsters' tactics, communication methods, victim details, location and time of occurrence. The collected data is stored in the server's internal database.

[1669] Training an AI model

[1670] Based on the collected data, the server trains an artificial intelligence model to learn fraud patterns, which includes criteria and algorithms to determine the likelihood of fraud.

[1671] Fraud judgment

[1672] The server analyzes the text data of the incoming call sent from the device and uses an AI model to determine the possibility of fraud. If it determines that there is a high possibility of fraud, it proceeds to the next step.

[1673] Response Generation

[1674] If fraud is determined to be likely, the server generates a natural-sounding response in the style of an elderly person, which is used to continue the dialogue with the fraudster and extract further information. The generated response text is then sent to the device.

[1675] Data Sharing

[1676] Collect and organize information about the characteristics and identity of fraudsters, and share it with the relevant authorities (police, etc.) in real time. This information sharing will contribute to the identification and arrest of fraudsters.

[1677] Terminal handling

[1678] Receiving a call

[1679] The device will use an unused phone number and will always be available to receive calls. If a scammer calls, the device will automatically receive and answer the call.

[1680] Voice Recognition

[1681] The device records the fraudster's voice in real time and converts it into text using voice recognition technology, which is then sent to a server.

[1682] Voice conversion and transmission

[1683] The response text received from the server is converted into an elderly-sounding voice, and the conversation continues by replying to the fraudster with the converted voice.

[1684] emotion recognition

[1685] The emotion engine installed on the device analyzes the user's emotions during the conversation with the scammer. For example, emotions such as anger, anxiety, and tension can be detected from the scammer's voice. This emotional information is sent to the server and reflected in the response generation.

[1686] User (system administrator) processing

[1687] System Monitoring

[1688] Users can check the dashboard to monitor the overall system status and quickly respond to any abnormalities or errors that occur.

[1689] System Optimization

[1690] Users can adjust system parameters based on collected data and system performance data, and retrain AI models to adapt to new fraudulent schemes.

[1691] Specific examples

[1692] As a specific scenario, let's consider a situation where a scammer calls and says, "Your bank account has been compromised. We need your immediate attention."

[1693] Terminal handling:

[1694] The device automatically receives the call and records the audio, which is then converted into text using voice recognition technology and sent to a server.

[1695] Server Action:

[1696] The server analyzes the received text, determines it is likely a scam, and then generates an elderly-sounding response to the device: "Sorry, what do I do?"

[1697] Terminal handling:

[1698] The device converts the received response text into speech and responds to the fraudster, while simultaneously analyzing the emotions in the fraudster's voice and sending the emotional information to the server.

[1699] Server Action:

[1700] The server uses this emotional information to generate an appropriate response that reflects the fraudster's emotional state when generating the next response. For example, if the fraudster is feeling anxious, the server can use this to devise a response that will further encourage the fraudulent tactics.

[1701] In this way, the server, terminal, and user work together to communicate with fraudsters in order to prevent fraud before it happens. Furthermore, by combining it with an emotion engine, the system can analyze the emotions of fraudsters, enabling more effective dialogue and information gathering.

[1702] The processing flow will be explained below.

[1703] Step 1:

[1704] Data collection (server processing)

[1705] The server periodically collects fraud data from fraud databases on the Internet and other sources. This data includes information about fraudsters' tactics, communication methods, victim details, location and time of occurrence. This collected data is stored in an internal database.

[1706] Step 2:

[1707] AI model training (server processing)

[1708] The server uses the collected data to retrain its artificial intelligence model, which learns fraud patterns, allowing it to adapt to the latest fraud techniques. The trained AI model is then used to identify fraud and generate responses for the next transaction.

[1709] Step 3:

[1710] Waiting for and receiving calls (terminal processing)

[1711] The device will use an unused phone number and will always be available to receive calls. If a scammer calls, the device will automatically answer the call and initiate the call.

[1712] Step 4:

[1713] Speech recognition and text conversion (device processing)

[1714] The device records the fraudster's voice in real time and converts this voice data into text using speech recognition technology, which is then sent to a server.

[1715] Step 5:

[1716] Fraud detection (server processing)

[1717] The server analyzes the text data sent from the device and uses an AI model to determine the likelihood of fraud. If it determines that there is a high possibility of fraud, it proceeds to the next step.

[1718] Step 6:

[1719] Emotion recognition (device processing)

[1720] An emotion engine on the device analyzes the scammer's voice and determines their emotional state (e.g., anger, impatience, calm, etc.) This emotion data is sent to the server for use in generating subsequent responses.

[1721] Step 7:

[1722] Response generation (server processing)

[1723] The server generates appropriate responses in the style of an elderly person based on the fraudster's tactics and emotional data. The responses are intended to continue the dialogue with the fraudster and elicit further information. The generated response text is then sent to the device.

[1724] Step 8:

[1725] Audio conversion and transmission (terminal processing)

[1726] The device uses speech synthesis technology to convert the response text received from the server into an elderly-sounding voice, which is then sent back to the fraudster, continuing the dialogue.

[1727] Step 9:

[1728] Information collection (server processing)

[1729] The server analyzes the information collected through interactions with the fraudsters and stores their characteristics (phone number, tactics, emotional state, etc.) in a comprehensive database.

[1730] Step 10:

[1731] Information sharing (server processing)

[1732] The server then shares the collected and analyzed information on fraudsters with the police and other relevant authorities in real time. This information sharing process facilitates the identification and arrest of fraudsters.

[1733] Step 11:

[1734] System monitoring (user actions)

[1735] Users can use a dashboard to monitor the overall system operation status and respond quickly to any errors or abnormalities that occur. Daily monitoring maintains the reliability and stability of the system.

[1736] Step 12:

[1737] System optimization (user processing)

[1738] Users can analyze the system's performance data and adjust configuration parameters as needed, including instructing the AI ​​model to retrain to adapt to new fraudulent schemes.

[1739] In a specific scenario, if a fraudster calls and says, "Your bank account is at risk. Immediate action is required," the system works as follows: The device receives the call and uses voice recognition technology to convert the content into text and send it to the server. The server analyzes the text, determines whether it is likely to be fraudulent, generates a response text, and sends it to the device. The device then converts the response text into speech and replies to the fraudster. During this process, the emotion engine analyzes the fraudster's emotions, and this data is reflected in the generation of the next response.

[1740] Example 2

[1741] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1742] In recent years, fraud has become more sophisticated and diverse, making it difficult to respond to conventional countermeasures. Telephone fraud targeting the elderly has been increasing in particular, and effective countermeasures are needed. Systems are needed that not only prevent fraud from occurring, but also collect and share information in real time to identify and arrest fraudsters. Furthermore, there is a need to expose fraudulent activities by analyzing fraudsters' emotions and engaging in more effective dialogue.

[1743] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes means for collecting data related to fraud, means for training an artificial intelligence model to learn fraud patterns based on the collected data, means for receiving communications from fraudsters and converting the content of the communications into text, means for determining the possibility of fraud using the converted text, means for generating a dialogue response if there is a possibility of fraud, means for converting the generated dialogue response into voice and replying to the fraudster, means for analyzing the characteristics of the fraudster and obtaining emotional information, and means for sharing information about the fraudster with relevant agencies. This improves the accuracy of fraud countermeasures, prevents fraudulent acts, and enables the collection and sharing of information to identify and arrest fraudsters in real time.

[1744] "Fraud Data" means information relating to fraudulent activity, including information such as the fraudulent method, means of communication, details of the victim, and the location and time of the occurrence.

[1745] An "artificial intelligence model" refers to an algorithm that learns from data and automatically performs specific tasks.

[1746] "Communications" refers to the means of communication used by scammers, such as voice or messages.

[1747] "Means for converting to text" refers to speech recognition technology for converting voice or messages into text data.

[1748] "Means for determining likelihood of fraud" refers to algorithms or models that analyze the converted text data and assess the risk of fraud.

[1749] "Means for generating dialogue responses" refers to an algorithm or generative model for automatically generating appropriate responses to continue the dialogue with the fraudster.

[1750] "Means for converting to speech" refers to speech synthesis technology for playing the generated text dialogue response as speech.

[1751] "Means of obtaining emotional information" refers to technology that analyzes emotions from the fraudster's voice and messages and identifies emotional states such as anger or impatience.

[1752] "Means of sharing information with relevant agencies" refers to communication methods and interfaces for providing information about fraudsters to relevant agencies, such as the police, in real time.

[1753] This invention is a system designed to prevent fraud by receiving calls from fraudsters, analyzing them, and strengthening countermeasures. The system consists of three main components: a server, a terminal, and a user, which work together.

[1754] Server Processing

[1755] Data collection

[1756] The server collects fraud data from online fraud databases and other sources, including fraudster tactics, communication methods, victim details, location and time of occurrence. Specifically, it retrieves data from the ScamWatch and FraudLog databases using HTTP requests and stores it in the server's internal database.

[1757] Training an AI model

[1758] The collected data is preprocessed and an artificial intelligence model is trained. For example, TensorFlow or PyTorch is used to learn fraud patterns. Specific features such as specific methods and time periods are incorporated into the algorithm to improve the accuracy of fraud detection.

[1759] Analysis of incoming calls and fraud detection

[1760] The server receives the text data of the incoming call sent from the device and analyzes it using an AI model. The analysis uses keywords and phrases contained in the text data to determine the possibility of fraud. For example, phrases such as "bank account," "risk," and "action required" are determined to indicate a high risk of fraud.

[1761] Response Generation

[1762] If fraud is deemed likely, the server uses a generative AI model (e.g., OpenAI GPT-4) to generate natural-sounding responses in the style of an elderly person, designed to continue the dialogue with the fraudster and elicit further information, such as "I'm sorry, what should I do?"

[1763] Data Sharing

[1764] The characteristics and identifying information of fraudsters are collated and shared with the relevant authorities (e.g., police) in real time. This information is sent via API to expedite response.

[1765] Terminal handling

[1766] Receiving a call

[1767] The device is set to an unused phone number and is always ready to receive calls. When a call comes in from a scammer, it automatically receives the call and prepares to record it. Specifically, it uses Twilio's call receiving API.

[1768] Speech recognition and text conversion

[1769] The device records the fraudster's voice in real time and converts it into text data using Google Cloud Speech-to-Text. The recorded audio stream is sent to an API, which outputs the text data, which is then sent to the server.

[1770] Voice conversion and transmission

[1771] The response text received from the server is converted into an elderly-sounding voice. Specifically, speech synthesis is performed using Amazon Polly or Google Text-to-Speech. The converted voice is then sent back to the fraudster, and the dialogue continues.

[1772] emotion recognition

[1773] The device uses the Microsoft Azure Emotion API to analyze emotional information from the fraudster's voice, detecting emotions such as anger, worry, and tension, and sending this emotional information to a server.

[1774] User (system administrator) processing

[1775] System Monitoring

[1776] Users use Grafana and Kibana to monitor the overall system status. If an abnormality or error occurs, they receive notifications and can respond quickly. They can understand the situation by checking log data and real-time notifications.

[1777] System Optimization

[1778] Users can adjust system parameters based on collected data and system performance data, and retrain the AI ​​model as needed to update the system to adapt to new fraudulent schemes.

[1779] Specific examples

[1780] As a specific scenario, let's consider a situation where a scammer calls and says, "Your bank account has been compromised. We need your immediate attention."

[1781] Terminal handling

[1782] The device automatically receives the call using Twilio, then converts the fraudster's speech into text in real time using Google Cloud Speech-to-Text, which then sends the text data to the server via an RTP stream.

[1783] Server Processing

[1784] The server receives the text data, "Your bank account is at risk. Immediate action is required." and analyzes it using an AI model. If it determines there is a high possibility of fraud, it uses OpenAI's GPT-4 to generate an elderly-sounding response, "I'm sorry, what should I do?", and sends it to the device.

[1785] Terminal handling

[1786] The device uses Amazon Polly to convert the received response text into an elderly-sounding voice and responds to the fraudster. At the same time, it uses Microsoft Azure's Emotion API to analyze emotional information (such as impatience or tension) from the fraudster's voice and sends it to the server.

[1787] Server Processing

[1788] The server will use this emotional information to generate the next response. For example, if it determines that the fraudster is anxious, it can generate a question like, "I'd like to confirm again, is your account really at risk?" This can further refine the fraudulent scheme.

[1789] Example prompts to input to the generative AI model

[1790] "Please translate the following sentence into a natural response from a senior citizen: 'Your bank account is at risk. Action is required immediately.'"

[1791] This prompt can be used to automatically generate an appropriate response.

[1792] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1793] Server Processing

[1794] Step 1: Data collection

[1795] The server collects fraud data from fraud databases and other sources on the internet. Specifically, it uses HTTP requests to retrieve information from the database about the fraud scheme, communication method, victim details, location and time of the incident. The input is the database API endpoint, and the output is detailed data about the fraud. The retrieved data is stored in the server's internal database.

[1796] Step 2: Training the AI ​​model

[1797] The server preprocesses the collected data and trains an artificial intelligence model. Specifically, it uses TensorFlow and PyTorch to learn fraud patterns. The input is the preprocessed dataset, and the output is a trained AI model. Features such as specific methods and time periods are reflected in the algorithm to improve the model's accuracy.

[1798] Step 3: Analyze incoming calls and identify fraud

[1799] The server receives the text data of the incoming call sent from the device and analyzes it using an AI model. The input is text data, and the output is a judgment result indicating the possibility of fraud. In this process, keywords and phrases contained in the text are analyzed to assess the risk of fraud.

[1800] Step 4: Response Generation

[1801] If the server determines that there is a high possibility of fraud, it uses a generative AI model to generate a natural-sounding response in the style of an elderly person. The input is the judgment result and the fraudster's text data, and the output is the generated response text. For example, it generates a response such as, "I'm sorry, what should I do?"

[1802] Step 5: Data sharing

[1803] The server compiles the characteristics and identifying information of fraudsters and shares it with the relevant authorities in real time. The input is detailed information about the fraudsters, and the output is the information sent to the relevant authorities. The information is sent quickly through the API, contributing to the identification and arrest of fraudsters.

[1804] Terminal handling

[1805] Step 1: Receiving a call

[1806] The terminal is set to an unused phone number and is always ready to receive calls. The input is a call from a fraudster, and the output is the received voice data. Specifically, it uses Twilio's call receiving API to automatically receive calls.

[1807] Step 2: Speech recognition and text conversion

[1808] The device records the fraudster's voice in real time and converts it into text data using Google Cloud Speech-to-Text. The input is audio data and the output is text data. The recorded audio stream is sent to the API, and the converted text is sent to the server.

[1809] Step 3: Convert and send

[1810] The device converts the response text received from the server into an elderly-sounding voice. The input is the response text, and the output is synthesized voice data. Specifically, Amazon Polly and Google Text-to-Speech are used to generate the voice and respond to the scammer.

[1811] Step 4: Emotion Recognition

[1812] An emotion engine implemented on the device is used to analyze emotions from the fraudster's voice. The input is the fraudster's voice data, and the output is emotional information (e.g., anger, tension, etc.). The emotional information is sent to the server and reflected in the generation of the next response.

[1813] User (system administrator) processing

[1814] Step 1: System monitoring

[1815] Users check a dashboard to monitor the overall system operation status. The input is real-time system log data, and the output is the monitoring results. Specifically, Grafana and Kibana are used to monitor and receive notifications for abnormalities and errors.

[1816] Step 2: System optimization

[1817] Users adjust the system parameters based on collected data and system performance data. The input is performance data and information on new fraud schemes, and the output is optimized system settings. If necessary, the AI ​​model can also be retrained.

[1818] (Application example 2)

[1819] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1820] The number of fraudulent phone calls, often involving sophisticated tactics by scammers, is steadily increasing, with elderly people being particularly vulnerable to these attacks. Current systems for combating fraudulent phone calls often struggle to identify these calls, resulting in delayed effective responses. Furthermore, they lack the means to analyze the emotions and psychological state of fraudsters, making it difficult to effectively deter and collect information from them. This can result in investigative authorities being unable to respond quickly, potentially exacerbating the damage. Therefore, there is a need for a system that can improve the accuracy and effectiveness of countermeasures against fraudulent phone calls by recognizing the emotions of fraudsters and generating responses that sound like an elderly person.

[1821] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1822] In this invention, the server includes means for collecting data on fraud, means for training an artificial intelligence model that learns fraud patterns based on the collected data, means for determining the possibility of fraud using the converted text, means for generating an elderly-sounding dialogue response if there is a possibility of fraud, means for converting the generated dialogue response into voice and replying to the fraudster, means for analyzing emotions from the fraudster's voice, and means for collecting information on the fraudster and sharing it with relevant institutions. This makes it possible to provide an appropriate response according to the fraudster's emotional state, effectively continue the dialogue to collect information, and ultimately prevent fraud before it occurs.

[1823] "Fraud Data" means any information relating to fraudulent activity, including the method of fraud, means of communication, details of the victim, location and time of the occurrence.

[1824] An "artificial intelligence model" refers to a computer model that can learn from large amounts of data and identify and analyze specific patterns and features.

[1825] "Convert to text" refers to the process of converting audio data into text data.

[1826] "Means for determining the likelihood of fraud" refers to the process of using artificial intelligence models or other algorithms to analyze and determine whether fraud is suspected.

[1827] "Elderly-style dialogue responses" refers to text and voice responses generated to make it appear as if an elderly person is having a conversation.

[1828] "Convert to voice" refers to the process of converting text data into voice data.

[1829] "Means for analyzing emotions from fraudsters' voices" refers to technology for identifying the emotional state (e.g., anger, impatience, tension, etc.) of fraudsters from their vocalizations.

[1830] "Fraudster information" refers to any information that can be used to identify a fraudster, including voice characteristics, phone number, words used, and conversation content.

[1831] "Means of sharing with relevant agencies" refers to the process for providing collected and analyzed fraud information to investigative agencies such as the police and monitoring agencies.

[1832] The present invention is a system for dealing with fraudulent calls from fraudsters, and is composed of a server, a terminal, and a user. The processing of the entire system will be described in detail below.

[1833] Server Processing

[1834] Data collection

[1835] The server periodically collects fraud data from online fraud databases and other sources. This data includes information about the fraud scheme, means of communication, victim details, location and time of occurrence. The collected data is stored in the server's internal database.

[1836] Training an AI model

[1837] Based on the collected data, the server trains an artificial intelligence model to learn fraud patterns, specifically by using pattern recognition techniques and machine learning algorithms to extract and learn the characteristics of suspicious communications.

[1838] Fraud judgment

[1839] The server analyzes the text data of the incoming call sent from the terminal and uses an artificial intelligence model to determine the possibility of fraud. If the result of this determination indicates a high possibility of fraud, the system proceeds to the next step.

[1840] Response Generation

[1841] If the server determines that fraud is likely, it generates a natural response in the style of an elderly person. This response is used to continue the dialogue with the fraudster and extract further information. The generated response text is sent to the device.

[1842] Data Sharing

[1843] Collecting characteristics and identifying information of fraudsters and sharing it with relevant authorities (such as the police) in real time, which will contribute to identifying and arresting fraudsters.

[1844] Terminal handling

[1845] Receiving a call

[1846] The device will always be ready to receive calls, and if a scammer calls you, it will automatically receive and answer the call.

[1847] Voice Recognition

[1848] The device records the fraudster's voice in real time and converts it into text using voice recognition technology (e.g., Google Cloud Speech-to-Text), which is then sent to a server.

[1849] Voice conversion and transmission

[1850] The response text received from the server is converted into an elderly-sounding voice. The converted voice is then sent back to the fraudster, continuing the dialogue. Tools such as Google Text-to-Speech (gTTS) are used for voice synthesis.

[1851] emotion recognition

[1852] An emotion engine (e.g., IBM Watson Tone Analyzer) installed on the device analyzes the emotions of the fraudster during the conversation. The analysis results are sent to the server and reflected in the response generation.

[1853] User (system administrator) processing

[1854] System Monitoring

[1855] Users can check the dashboard to monitor the overall system status, and respond quickly if an abnormality or error occurs.

[1856] System Optimization

[1857] Users can adjust system parameters based on collected data and system performance data, and retrain AI models to adapt to new fraudulent schemes.

[1858] Specific examples

[1859] For example, imagine a scammer calling you and saying, "Your bank account is at risk. We need your immediate attention."

[1860] 1. Terminal Processing

[1861] The device automatically receives the call and records the audio, which is then converted into text using voice recognition technology and sent to a server.

[1862] 2. Server Processing

[1863] The server analyzes the received text and determines that it is likely a scam, then generates an elderly-sounding response to the user, such as "Really? What should I do?", and sends it to the device.

[1864] 3. Terminal Processing

[1865] The device converts the received response text into speech and responds to the fraudster. It also analyzes the emotions in the fraudster's voice and sends the emotional information to the server.

[1866] 4. Server Processing

[1867] The server then uses the emotional information to generate an appropriate response that reflects the fraudster's emotional state. For example, if the fraudster is feeling anxious, the server can use this to devise a response that will further encourage the fraudulent tactics.

[1868] Example prompt sentence:

[1869] "Pretend to be a senior citizen and respond naturally to the following statement: 'Your bank account is at risk.'"

[1870] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1871] Step 1:

[1872] Receiving a call (terminal)

[1873] The device is always ready to receive calls. When a call comes in from a fraudster, it automatically answers the call and starts the conversation. The input is the call signal from the fraudster, and the output is the call audio data.

[1874] Step 2:

[1875] Voice recording and recognition (device)

[1876] The device records the scammer's voice in real time and converts the voice data into text data using voice recognition technology such as Google Cloud Speech-to-Text. The input is the recorded voice data, and the output is text data.

[1877] Step 3:

[1878] Sending text data (terminal → server)

[1879] The terminal sends the converted text data to the server. The input is the text data generated by the speech recognition technology, and the output is the text data sent to the server.

[1880] Step 4:

[1881] Fraud detection (server)

[1882] The server analyzes the received text data and uses an artificial intelligence model (generative AI model) to determine the likelihood of fraud. The input is text data, and the output is a determination result indicating the likelihood of fraud. This data is processed through data calculations and analysis within the server.

[1883] Step 5:

[1884] Generate response (server)

[1885] If the server determines that there is a high possibility of fraud, it generates a natural response in the style of an elderly person for the next dialogue step, where it uses a generative AI model and a prompt to generate an appropriate response. The input is the fraud detection result and the prompt, and the output is the generated response text.

[1886] Step 6:

[1887] Sending a response (server → terminal)

[1888] The server sends the generated response text to the terminal. The input is the generated response text, and the output is the text data sent to the terminal.

[1889] Step 7:

[1890] Voice conversion and reply (terminal)

[1891] The device converts the received response text into an elderly-sounding voice and generates an audio file using Google Text-to-Speech (gTTS), which then replies to the fraudster using this generated voice. The input is the response text, and the output is the voice response to the fraudster.

[1892] Step 8:

[1893] Emotion Recognition (Device)

[1894] The device analyzes the emotions from the fraudster's voice and sends the results to a server, using technologies such as IBM Watson Tone Analyzer. The input is the fraudster's voice data, and the output is the analyzed emotion data.

[1895] Step 9:

[1896] Receiving and responding to emotion data (server)

[1897] The server receives the emotion data sent from the device and reflects it when generating the next response based on that emotion information. This makes it possible to generate an appropriate response based on the fraudster's psychological state. The input is emotion data, and the output is the next response text based on the emotion.

[1898] Step 10:

[1899] Fraudster information collection and sharing (server)

[1900] The server organizes and collects information obtained from fraudsters and shares it with the relevant authorities (police, etc.) in real time. The input is the fraudster information obtained from the conversation, and the output is the shared information sent to the relevant authorities.

[1901] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1902] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1903] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1904] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1905] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1906] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1907] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1908] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1909] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1910] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1911] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1912] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1913] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1914] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1915] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1916] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1917] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1918] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1919] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1920] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1921] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1922] The following is further disclosed regarding the above embodiment.

[1923] (Claim 1)

[1924] means of collecting data on fraud;

[1925] means for training an artificial intelligence model to learn fraud patterns based on the collected data; and

[1926] A means of receiving calls from scammers and converting them into text;

[1927] means for determining likelihood of fraud using the converted text;

[1928] a means for generating an interactive response in the event of potential fraud;

[1929] a means for converting the generated dialogue response into speech and replying to the fraudster;

[1930] A means of collecting information on fraudsters and sharing that information with relevant authorities;

[1931] A system including:

[1932] (Claim 2)

[1933] 10. The system of claim 1, further comprising means for automatically extracting and storing characteristic information of the fraudster during an interaction with the fraudster.

[1934] (Claim 3)

[1935] ...

Claims

1. means of collecting data on fraud; means for training an artificial intelligence model to learn fraud patterns based on the collected data; and A means of receiving calls from scammers and converting them into text; means for determining likelihood of fraud using the converted text; a means for generating an interactive response in the event of potential fraud; a means for converting the generated dialogue response into speech and replying to the fraudster; A means of collecting information on fraudsters and sharing that information with relevant authorities; A system including:

2. 10. The system of claim 1, further comprising means for automatically extracting and storing characteristic information of a fraudster during an interaction with the fraudster.

3. 10. The system of claim 1, further comprising means for analyzing the collected data and characteristic information to detect new fraudulent schemes and propose countermeasures.

Images