system

Abstract

We provide the system. [Solution] A means of analyzing audio data received by a communication device to determine the possibility of fraudulent activity, Based on the determination result, the system activates an automated response device and generates a response using natural language processing technology. A means for outputting the generated response content using speech synthesis technology, A means of recording conversation data with scammers and saving it in a database, A means of learning new patterns of fraudulent activity based on saved conversation data, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Fraudulent calls targeting the elderly have become more sophisticated year by year, and the amount of damage has increased rapidly. In particular, the elderly are likely to be information disadvantaged and are often targeted by fraudsters. Conventional countermeasures have mainly focused on preventing fraudulent calls and promoting vigilance, but it is difficult to effectively repel fraudsters with just that. Therefore, a new mechanism for protecting the elderly while exhausting fraudsters is required.

Means for Solving the Problems

[0005] This invention provides a means for analyzing voice data received by a communication device and determining the possibility of fraudulent activity. Based on this determination, an automated response device is activated, and conversation content with the fraudster is generated using natural language processing technology. The generated content is output using speech synthesis technology, exhausting the fraudster by continuing a meaningless conversation. Furthermore, all conversation data is recorded and stored, and by learning new patterns of fraudulent activity based on the obtained data, the AI ​​model can perform more accurate identification of fraudulent calls. This system can prevent elderly people from becoming victims of fraudulent calls.

[0006] A "communication device" is an electronic device that has the function of receiving and processing voice data.

[0007] "Voice data" refers to voice information acquired through communication devices during a phone call.

[0008] "Fraudulent activity" refers to unethical acts that attempt to deceive others and obtain unfair profits.

[0009] An "automatic response device" is a device that automatically generates and outputs a response in response to a specific trigger.

[0010] "Natural language processing technology" is the technology that enables computers to understand and generate human language.

[0011] "Speech synthesis technology" is a technology that converts text data into speech information.

[0012] A "database" is an information management system that organizes and stores large amounts of information, making it accessible as needed.

[0013] An "AI model" is a problem-solving algorithm built through the learning process of artificial intelligence. [Brief explanation of the drawing]

[0014] [Figure 1]It is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] It is a conceptual diagram showing an example of the main functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Embodiments for Carrying Out the Invention

[0015] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0016] First, the terms used in the following description will be explained.

[0017] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0018] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0019] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0020] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0021] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0022] [First Embodiment]

[0023] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0024] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0025] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0026] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0027] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0028] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0029] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0030] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0031] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0032] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0033] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0034] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0035] The fraudulent call prevention system of the present invention consists of a communication device, a server, a user terminal, and a program that links these components. The user terminal receives an incoming call, performs initial processing, and sends the information to the server. The server analyzes the received phone number and voice data to determine the possibility of fraudulent activity.

[0036] If fraudulent activity is detected, the server, through an automated response system, speaks a conversation generated using natural language processing technology to the fraudster. The terminal then uses speech synthesis technology to convert this generated response into speech and processes it to allow the call with the fraudster to continue.

[0037] For example, if a user's device receives a call and the number is on a past list of suspicious numbers, the server immediately determines that it is highly likely to be a scam. The server activates an automated answering system and generates a natural-sounding response, such as, "I can't hear you very well, could you please repeat your name?" This synthesized response is spoken to the scammer in real time.

[0038] All interactions with scammers are recorded on the server, and an AI model later reviews this data to learn new scam patterns. This system ensures that users are safely protected from the direct impact of scam calls. Furthermore, the recorded data contributes to overall system improvements and enhances the system's ability to detect scams more effectively.

[0039] By implementing this invention, it is possible to significantly reduce the damage caused by telephone scams targeting the elderly, and it is expected that this will improve social safety.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The device detects an incoming call. It sends a message to the server, along with audio data, indicating that an incoming call has been received.

[0043] Step 2:

[0044] The server receives the incoming phone number and voice data and determines whether it is likely to be a scam. By comparing it with a stored database of past scams, it determines whether or not it is likely to be a scam.

[0045] Step 3:

[0046] Based on the server's assessment, if it determines there is a high probability of fraud, it activates an automated response system. Natural language processing technology is used to generate responses that mimic those of an elderly person.

[0047] Step 4:

[0048] The server sends the generated response to the terminal. This response might be something like, "Could you please repeat that slowly?"

[0049] Step 5:

[0050] The terminal receives a response from the server, converts it into speech using speech synthesis technology, and speaks it to the other party. To ensure the speech doesn't sound unnatural, it maintains a natural tone, mimicking that of an elderly person.

[0051] Step 6:

[0052] The server records all conversations with the scammer. Repeat steps 3 through 5 as long as the conversation continues.

[0053] Step 7:

[0054] Once the conversation ends, the server saves the recorded data to a database. This data is used to train AI models, improving their accuracy in responding to new fraudulent schemes.

[0055] Step 8:

[0056] The server notifies the user that a potentially fraudulent call was received and that the matter has been resolved. A summary of the conversation is reported if necessary.

[0057] (Example 1)

[0058] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0059] In light of the increasing prevalence of telephone-based fraud, this invention aims to effectively protect ordinary users, particularly the elderly, from becoming victims of fraud. It seeks to provide an environment where users can use telephone communication with peace of mind, without being directly affected by fraudulent calls. Furthermore, developing technology capable of responding quickly and accurately to increasingly diverse and sophisticated fraudulent activities is also a crucial objective.

[0060] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0061] In this invention, the server includes means for analyzing voice information received by a communication device and evaluating the probability of fraudulent activity, means for generating conversation content using natural language processing technology with a response generation device, and means for outputting the generated conversation content using speech synthesis technology. This makes it possible to detect fraudulent calls more quickly and accurately than with conventional manual responses, protect users from fraud, and ensure user safety.

[0062] "Communication equipment" refers to devices used to send and receive voice and data, and includes telephones, smartphones, and other similar devices.

[0063] "Voice information" refers to the voice data exchanged during a phone call, which is recorded as human voice.

[0064] "Analysis" refers to processing data and extracting information, and in particular, includes the process of converting audio data into text and evaluating it.

[0065] "Assessing the probability of fraud" is the process of numerically or qualitatively estimating the likelihood of fraud based on the information received.

[0066] A "response generation device" is a device that has the function of generating an appropriate response in natural language based on received audio information and analysis results.

[0067] "Natural language processing technology" refers to the technology that enables computers to understand, process, and generate human language.

[0068] "Speech synthesis technology" is a technology that outputs text information as speech, generating sound waveforms that closely resemble human voices.

[0069] "Dialogue information" refers to the content of conversations recorded during a phone call, which is stored for later analysis and learning.

[0070] "Storage device" refers to a device or system for storing data, and includes databases and cloud storage.

[0071] "Learning" refers to the process by which a machine learning model extracts knowledge and patterns using new and historical data, thereby improving its accuracy and performance.

[0072] A "database" is a collection of data that has a structure that organizes information and allows for efficient access.

[0073] "Immediately assessing the degree of suspicion" refers to the process of quickly determining the reliability of the received communication number and content, and immediately estimating the possibility of fraud.

[0074] The present invention involves deploying a system that combines communication equipment, servers, and user terminals to effectively protect users from fraudulent phone calls. It primarily implements functions such as voice information analysis, fraudulent activity evaluation, automated response generation, speech synthesis output, and fraud pattern learning.

[0075] The server utilizes speech recognition and natural language processing technologies to process audio information received through communication devices. Specifically, the server uses speech recognition software (e.g., a common speech recognition API) to convert audio into text and then evaluates the likelihood of fraudulent activity based on that text. This evaluation employs machine learning models to compare the received phone number with past suspicious cases registered in the database.

[0076] The user terminal receives instructions from the server and automatically responds. The server uses a generative AI model to generate natural-sounding dialogue based on the prompt text. An example of a prompt text might be, "Generate a response that will reassure an elderly person in a fraudulent phone call." This generated response is output as actual voice on the user terminal using speech synthesis technology. An existing speech synthesis engine is used for speech synthesis.

[0077] Such a system allows, for example, elderly users who become targets of phone scams to deal with the situation safely without having to come into contact with the scammers. All conversational information is recorded on a server, and this data is used for the continuous learning of the generating AI model. This improves the overall fraud detection capability of the system and allows it to quickly adapt to new fraud patterns.

[0078] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0079] Step 1:

[0080] When a user terminal receives an incoming call, it obtains the phone number and caller information and sends it to the server. This input includes the received phone number and voice data. The user terminal encrypts this data and sends it to the server. This ensures the protection of personal information while enabling rapid analysis.

[0081] Step 2:

[0082] The server analyzes the received audio data. The input audio data is converted into text data using speech recognition technology. Next, the server compares the phone number with the database and evaluates whether it falls under the suspicious list. This comparison allows for an immediate determination of the degree of suspicion and an assessment of the possibility of fraudulent activity.

[0083] Step 3:

[0084] If the server determines that a call is likely to be a scam, it sends a prompt to the AI ​​model. The prompt might say, "Generate a reassuring response for an elderly person in a scam call," and the AI ​​model generates a natural language dialogue. This generation process is optimized based on past data and similar cases.

[0085] Step 4:

[0086] The generated response is sent from the server to the user's terminal, which outputs it as actual voice using speech synthesis technology. This output uses speech synthesis that has a sound quality close to that of a human voice. The terminal continues to interact with the scammer, adjusting the volume and tone to ensure the conversation proceeds naturally.

[0087] Step 5:

[0088] The server records the content of conversations with scammers and stores it in a database. The recorded conversation information is used in subsequent learning processes. The server updates the generated AI model based on the stored data and continues to learn in order to detect new scam patterns. This improves the overall scam detection capability of the system.

[0089] (Application Example 1)

[0090] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0091] Traditional fraud prevention systems often only detect fraudulent activity using fixed methods, limiting their ability to combat the ever-evolving nature of fraud. Furthermore, these systems require frequent updates, leading to user-intensive operation and difficulties in responding immediately to new fraud techniques. Additionally, insufficient awareness campaigns about fraudulent calls targeting the elderly, coupled with the inability to respond in real-time, raise concerns about the potential for widespread victimization.

[0092] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0093] In this invention, the server includes means for analyzing voice data received by a communication device and determining the possibility of fraudulent activity, means for activating an automated response device and generating a response using natural language processing technology, and means for outputting the generated response using speech synthesis technology. This enables real-time detection and immediate response to fraudulent activity, ensuring safety for the elderly and other users. Users can receive immediate fraud warnings on their devices through a pre-installed mobile application. Furthermore, by linking with cloud services, it is possible to respond quickly to various fraud patterns, reduce the complexity of user operation, and flexibly respond to the latest fraudulent activities.

[0094] A "communication device" is a device that receives voice data, analyzes that data, and has the function of determining the possibility of fraudulent activity.

[0095] An "automated response device" is a device that can speak responses generated using natural language processing technology to fraudsters.

[0096] "Natural language processing technology" is a technology for analyzing, understanding, and generating human language, enabling smooth communication between machines and humans.

[0097] "Speech synthesis technology" is a technology that converts text data into speech data and outputs it as a natural-sounding human voice.

[0098] A "database" is a storage device for saving conversation data with scammers, allowing for quick retrieval and reference of data as needed.

[0099] A "mobile application" is software that runs on a user's mobile device to display the results of fraud detection and provide warnings.

[0100] "Cloud services" utilize computing resources and data storage provided via the internet, and serve as a foundation for rapidly transmitting and updating the results of fraud detection.

[0101] A "user interface" is an interface that allows a user to interact with a computer system, and is designed for displaying and inputting information.

[0102] The implementation of this invention primarily involves a system in which a server, a terminal, and a user cooperate to prevent fraudulent phone calls. In its specific form, the invention begins with a step in which a communication device analyzes received voice data to determine the possibility of fraudulent activity. The server utilizes a communication device for this determination, comparing the received phone number and voice data with a database via a cloud service to evaluate the likelihood of fraud.

[0103] Based on the detection results, if fraudulent activity is detected, the server automatically activates an automated response system. This system uses natural language processing technology to generate an appropriate response and speaks it to the fraudster via speech synthesis technology. This allows for secure data collection while maintaining a call with the fraudster.

[0104] Simultaneously, a mobile application runs on the device, and when fraudulent activity is detected, the results are sent via a cloud service, and the user is warned through the user interface. This process allows users to quickly recognize fraudulent activity and prevent becoming a victim.

[0105] As a concrete example, let's assume a user has this system installed on their smartphone and receives a call. The system immediately identifies the number "080-XXXX-XXXX" as a scam and responds with an automated voice message saying, "I'm sorry, but could you please tell me your name again?" During this time, the user receives a real-time warning on their mobile device and can take the necessary action.

[0106] Examples of prompts for a generative AI model include the following:

[0107] A scam call has been identified. Please ask the scammer the following question: "Excuse me, but could you please tell me your name again?"

[0108] This embodiment allows for flexible responses to the latest fraudulent methods and provides users with peace of mind.

[0109] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0110] Step 1:

[0111] The server obtains voice data and phone numbers received from the user's terminal via a communication device. The voice data, as input data, is subjected to a voice analysis algorithm, and the phone numbers are compared against a database in the cloud. This process calculates the likelihood of fraudulent activity.

[0112] Step 2:

[0113] The server determines the likelihood of fraud based on the phone number and the results of voice analysis. If it determines that there is a high probability of fraud, it retrieves the relevant information from the database and generates a trigger to activate the automated response system. The input in this step is the phone number and the analysis results, and the output is the fraud determination result.

[0114] Step 3:

[0115] The device receives the fraud detection result from the server and notifies the user through the mobile application. The application displays a warning message using the user interface and prompts the user to take action. The input is the detection result from the server, and the output is the warning display.

[0116] Step 4:

[0117] If the server determines that a user is a scammer, it uses an automated response system to generate a prompt message to respond to the scammer. This prompt message is generated using natural language processing technology and output in real time using speech synthesis technology. The input is the judgment result and the generating AI model, and the output is the synthesized response.

[0118] Step 5:

[0119] The server records the call data with the scammer and stores it in a database so that the AI ​​model can learn new scam patterns at a later date. In this step, the call data is used as input, and the output is the stored training data.

[0120] These processing steps enable the system to detect fraud in real time and notify users, providing a systematic approach to combating fraudulent activity.

[0121] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0122] The system of the present invention is comprised of a communication device, a server, a user terminal, and an emotion engine. The user terminal first detects an incoming call and transmits the corresponding voice data to the server and the emotion engine. The server captures the incoming number and voice data and determines the possibility of fraudulent activity by comparing it with a database. Based on this determination, it activates an automated response system and generates an appropriate response using natural language processing technology.

[0123] Furthermore, the emotion engine analyzes the voice data to recognize the user's emotional state. It extracts the type and intensity of the emotion and sends it to the server. The server then takes this emotional information into account and adjusts the response accordingly. For example, if the user is showing anxiety or confusion, the response can be changed to a gentler tone to provide reassurance.

[0124] The generated response is converted into speech using speech synthesis technology and spoken to the other party. The server records all resulting interactions with the scammer and stores them in a database as more detailed data. This allows the AI ​​model to use the emotional responses as training data for learning scam patterns.

[0125] For example, when a user receives a fraudulent phone call, the server checks the number and recognizes the possibility of fraud. If the emotion engine determines from the user's voice that they are remaining calm, the server generates a normal response accordingly, resulting in a natural conversation. On the other hand, if the emotion engine determines that the user is showing signs of anxiety, it adjusts the response to provide reassurance, such as, "It's okay, let's talk slowly."

[0126] This system enhances protection against fraudulent activities and ensures the emotional safety of users. Through these processes, users can be effectively protected from fraudulent phone calls.

[0127] The following describes the processing flow.

[0128] Step 1:

[0129] The device detects an incoming call. It sends the incoming call information and voice data to the server and the emotion engine.

[0130] Step 2:

[0131] The server compares the incoming number against a database of past fraudulent numbers to determine the likelihood of fraud. If the result indicates a high probability of fraud, the automated response system activation flag is set.

[0132] Step 3:

[0133] The emotion engine analyzes the received audio data and extracts the user's emotional state. It generates data about the type and intensity of the emotion and sends it to the server.

[0134] Step 4:

[0135] The server receives emotional data from the emotion engine and uses natural language processing techniques to adjust the response. For example, if the user is expressing anxiety, the response is modified to be more reassuring.

[0136] Step 5:

[0137] The server sends the generated response to the terminal. The terminal then uses speech synthesis technology to convert this response into speech.

[0138] Step 6:

[0139] The device speaks a voiced response to the scammer. The voice tone and tempo are set to mimic an elderly person's voice, reducing any unnaturalness.

[0140] Step 7:

[0141] The server continuously records conversations with scammers and stores all data in a database. This includes conversation content and user sentiment data.

[0142] Step 8:

[0143] Once the server determines that the conversation has ended, it uses the recorded data as training material for the AI ​​model. This improves its ability to respond to new fraud patterns and emotional responses.

[0144] Step 9:

[0145] The server notifies the user when the conversation has ended and, if necessary, reports a summary to confirm that the scam call was handled properly.

[0146] (Example 2)

[0147] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0148] Conventional fraud prevention systems rely on simple methods such as number matching to identify fraud, making it difficult to detect when fraudsters change their numbers. Furthermore, they lack systems that take into account the emotional reactions of users, which hinders their ability to enhance user confidence.

[0149] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0150] In this invention, the server includes means for analyzing audio data received by a communication device and determining the possibility of fraudulent activity; means for activating an automated response device and generating a response using natural language processing technology based on the determination result; and means for outputting the generated response using speech synthesis technology. This makes it possible to highly detect fraudulent activity through audio data, and at the same time enhance the user's sense of security and strengthen the defense against fraudulent activity by providing a response that takes into account the user's emotional state.

[0151] A "communication device" is a device that has the function of sending and receiving voice data, and usually has an interface for exchanging data with other devices via a network.

[0152] "Audio data" refers to data used to record and process audio in digital format, and includes the content of a call and its voice patterns.

[0153] "Fraud detection" is a process of evaluating the likelihood of fraud based on received data and detecting such activity.

[0154] An "automatic response device" is a device that autonomously generates a response through a program and provides that response to the user or other party in real time.

[0155] "Natural language processing technology" is a technology that understands, analyzes, and generates human language, making it possible for machines to communicate with humans in a natural way.

[0156] "Speech synthesis technology" is a technology that converts text data into speech data and is used to generate natural-sounding speech.

[0157] "Emotional state" refers to a person's psychological state inferred from their voice or text, and includes emotions such as anxiety, relief, and anger.

[0158] "Response content" refers to the content of natural-sounding conversations and messages generated based on the received data and emotional state.

[0159] A "database" is a structured data storage system for efficiently storing, managing, and accessing information.

[0160] A "generative AI model" is an algorithm that uses machine learning techniques to learn patterns from large amounts of data and then makes predictions or generates new data.

[0161] The present invention is implemented as a system combining a communication device, a computer server, a user terminal, and an emotion engine. This system aims to detect fraudulent calls and optimize user responses, and is realized through the use of various technologies.

[0162] First, when the user terminal detects an incoming call, it collects voice data in real time and transmits it to a computer server and emotion engine via the internet. The user terminal uses a secure protocol to ensure the safe transfer of data.

[0163] Next, the server uses a specialized algorithm to analyze the voice data and the incoming number. To determine the likelihood of fraud, it performs number verification in conjunction with a database. If a number registered as a fraudulent number is detected, the server activates an automated response system. Using natural language processing technology, it generates a response based on the received information, and then uses speech synthesis technology to convert that response into speech.

[0164] The emotion engine uses a voice analysis algorithm to extract and recognize the user's emotional state in real time from voice data. For example, if anxiety is detected from the voice tone or speed, that emotional information is sent to the server. The server then adjusts the response based on that emotional information to provide the user with a reassuring response.

[0165] For example, when a user receives a scam call, the server is configured to respond in a gentle tone, such as "Don't worry, we'll support you," if the emotion engine reports that the user is feeling anxious.

[0166] An example of a prompt message would be, "If a user receives a fraudulent phone call, please tell me how to respond if they express anxiety." This is used to instruct the generating AI model on specific actions to take.

[0167] This system consists of communication devices, terminals, and specialized software running on servers, and functions to prevent fraudulent activities and enhance user confidence. In this way, users can handle phone calls more safely and with greater peace of mind.

[0168] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0169] Step 1:

[0170] Incoming call detection and data collection

[0171] The terminal detects incoming calls. Using a communication device API, it acquires the incoming number and voice data in real time. Inputs include the incoming call event, phone number, and voice signal, while output is the acquired voice data, prepared for transmission to the server and emotion engine.

[0172] Step 2:

[0173] Sending audio data

[0174] The terminal transmits the received voice data to the server and emotion engine via a secure communication protocol (e.g., HTTPS). Inputs include voice data and the incoming number, while output is the transmission of this data to the server and emotion engine.

[0175] Step 3:

[0176] Identifying fraudulent activity

[0177] The server uses the received voice data and incoming number to determine the possibility of fraud. It accesses a database and compares the incoming number against a list of known fraudulent numbers. The input is the incoming number and voice data, and the output is either a flag indicating the possibility of fraud or a status of "not found".

[0178] Step 4:

[0179] Analysis of emotional states

[0180] The emotion engine analyzes voice data to determine the user's emotional state. It analyzes voice tone, pitch, speed, etc., to extract emotional information. The input is voice data, and the output is the type of emotion (e.g., anxiety, relief) and its intensity.

[0181] Step 5:

[0182] Generating the response content

[0183] The server considers the presence or absence of fraudulent activity and sentiment information, and uses a generative AI model to create an appropriate response. The prompt in this case is in the form of, "If the user is showing anxiety, please generate an appropriate response." The input consists of a discrimination flag and sentiment data, and the output is the generated text of an optimized response.

[0184] Step 6:

[0185] Speech synthesis and responsive speech

[0186] The server uses speech synthesis technology to convert the generated response into speech and automatically plays it during the call with the scammer. The input is the generated text data, and the output is the generated audio data that is spoken to the other party on the call.

[0187] Step 7:

[0188] Data recording and storage

[0189] The server records and stores data in a database, including all interactions related to fraudulent activity and the emotional responses at the time. Inputs include conversation log data and emotional information, while output is this information stored in a structured format, making it available as subsequent training data.

[0190] (Application Example 2)

[0191] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0192] Conventional fraud prevention systems focus on identifying fraudulent activity through the analysis of voice data, but they have the drawback of lacking consideration for the user's emotional state. In particular, when a user experiences anxiety or confusion due to a fraudulent call, the system may not be able to respond appropriately, potentially compromising the user's emotional safety.

[0193] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0194] In this invention, the server includes means for analyzing audio information received by communication equipment and determining the possibility of fraudulent activity; means for analyzing the emotional state from the received audio information and adjusting the response content based on the user's emotions; and means for recording the conversation record with the fraudster in a storage device and utilizing it as a source of information for learning new fraudulent activity patterns. This makes it possible to strengthen the defense against fraudulent activity and ensure the emotional safety of the user.

[0195] "Communication equipment" is a general term for devices and terminals used to receive and analyze voice information.

[0196] "Audio information" refers to data captured by communication devices as audio signals, and is used as material for analysis and response generation.

[0197] A "discrimination method" is a system for evaluating and judging the possibility of fraudulent activity based on audio information.

[0198] An "automatic response device" is a system that generates an appropriate response based on the discrimination result.

[0199] "Natural language processing technology" is a technology that allows computers to understand and generate human language, and is used to generate response content.

[0200] "Emotional state" refers to the type and intensity of the user's emotions, as analyzed from their voice information.

[0201] "Response content" refers to the response to the user generated based on the analysis of the voice information.

[0202] "Speech synthesis technology" is a technology that converts generated responses into speech and outputs them.

[0203] A "conversation record" is data that records and saves the content of conversations with a scammer.

[0204] A "storage device" is a medium for storing conversation records and using them for later learning and analysis.

[0205] "Information sources" refer to data that provides the necessary information for learning about new fraudulent activities, and include conversation records, etc.

[0206] The system for carrying out this invention consists of a communication device, a server, an emotion recognition engine, an automatic response device, and a speech synthesis device. The communication device is responsible for receiving voice information from the user and transmitting it to the server. The server analyzes the received voice information and first determines the possibility of fraudulent activity. This determination is made by comparing the contact number and voice data with a database of past fraudulent activities. Furthermore, the server uses the emotion recognition engine to analyze the user's emotional state from the voice information. The emotion recognition engine evaluates the user's emotions based on parameters such as voice waveform, frequency, and speed.

[0207] The server then generates the optimal response that the automated response system should provide to the user, based on the likelihood of fraud and the user's emotional state. Utilizing natural language processing technology, the response is tailored to soothe the user's emotions. The generated response is then converted into natural-sounding speech by a speech synthesizer and returned to the user via communication equipment.

[0208] Furthermore, all conversation records are recorded and stored in a storage device. The stored data is used as information for the server to learn new fraud patterns. This allows the system to continuously improve the accuracy of fraud detection.

[0209] For example, if a user receives a suspicious call, the server immediately analyzes the audio information and, if it determines that there is a high probability of fraud, it retrieves data indicating that the user's emotions are unstable. In response, a reassuring response such as "It's okay, let's talk slowly?" is generated. An example of a prompt to the generating AI model would be, "Analyze the audio data of this call and assess the possibility of a fraudulent transaction. If the user is feeling anxious, generate an appropriate support message."

[0210] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0211] Step 1:

[0212] Communication equipment receives voice information from a user. The input is the user's spoken voice, and the output is digitized voice data. Voice sampling is performed to convert the voice into a digital signal.

[0213] Step 2:

[0214] The server receives the audio data and analyzes it to determine the possibility of fraudulent activity. The input is the audio data from step 1, and the output is the evaluation result of the fraud possibility. The server refers to the database, matches the audio data and contact numbers, and performs fraud pattern matching.

[0215] Step 3:

[0216] The server activates an emotion recognition engine and analyzes the user's emotional state from the audio data. The input is audio data, and the output is the user's emotion evaluation result. Audio parameters (waveform, audio frequency, speed, etc.) are calculated to identify the emotion.

[0217] Step 4:

[0218] The server applies natural language processing techniques to generate the optimal response based on the discrimination result and emotional state. The input is the fraud discrimination result and the emotional evaluation result, and the output is the generated text response. A generative AI model performs the language generation process.

[0219] Step 5:

[0220] A speech synthesizer converts the generated text response into speech. The input is the generated text response, and the output is the synthesized speech. A speech synthesis engine converts text into speech.

[0221] Step 6:

[0222] The server generates audio and returns it to the user via the communication device. The input is synthesized speech, and the output is the audio provided to the user. The communication device plays the audio.

[0223] Step 7:

[0224] The server saves conversation records and stores them in a database. Inputs are log information and audio recordings from all processes, and output is the conversation data stored in the database. A storage solution is used for data storage.

[0225] Step 8:

[0226] The server learns new fraud patterns using stored data. The input is conversation records in the database, and the output is an updated fraud detection model. A machine learning algorithm performs the learning process.

[0227] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0228] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0229] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0230] [Second Embodiment]

[0231] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0232] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0233] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0234] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0235] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0236] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0237] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0238] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0239] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0240] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0241] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0242] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0243] The fraudulent call prevention system of the present invention consists of a communication device, a server, a user terminal, and a program that links these components. The user terminal receives an incoming call, performs initial processing, and sends the information to the server. The server analyzes the received phone number and voice data to determine the possibility of fraudulent activity.

[0244] If fraudulent activity is detected, the server, through an automated response system, speaks a conversation generated using natural language processing technology to the fraudster. The terminal then uses speech synthesis technology to convert this generated response into speech and processes it to allow the call with the fraudster to continue.

[0245] For example, if a user's device receives a call and the number is on a past list of suspicious numbers, the server immediately determines that it is highly likely to be a scam. The server activates an automated answering system and generates a natural-sounding response, such as, "I can't hear you very well, could you please repeat your name?" This synthesized response is spoken to the scammer in real time.

[0246] All interactions with scammers are recorded on the server, and an AI model later reviews this data to learn new scam patterns. This system ensures that users are safely protected from the direct impact of scam calls. Furthermore, the recorded data contributes to overall system improvements and enhances the system's ability to detect scams more effectively.

[0247] By implementing this invention, it is possible to significantly reduce the damage caused by telephone scams targeting the elderly, and it is expected that this will improve social safety.

[0248] The following describes the processing flow.

[0249] Step 1:

[0250] The device detects an incoming call. It sends a message to the server, along with audio data, indicating that an incoming call has been received.

[0251] Step 2:

[0252] The server receives the incoming phone number and voice data and determines whether it is likely to be a scam. By comparing it with a stored database of past scams, it determines whether or not it is likely to be a scam.

[0253] Step 3:

[0254] Based on the server's assessment, if it determines there is a high probability of fraud, it activates an automated response system. Natural language processing technology is used to generate responses that mimic those of an elderly person.

[0255] Step 4:

[0256] The server sends the generated response to the terminal. This response might be something like, "Could you please repeat that slowly?"

[0257] Step 5:

[0258] The terminal receives a response from the server, converts it into speech using speech synthesis technology, and speaks it to the other party. To ensure the speech doesn't sound unnatural, it maintains a natural tone, mimicking that of an elderly person.

[0259] Step 6:

[0260] The server records all conversations with the scammer. Repeat steps 3 through 5 as long as the conversation continues.

[0261] Step 7:

[0262] Once the conversation ends, the server saves the recorded data to a database. This data is used to train AI models, improving their accuracy in responding to new fraudulent schemes.

[0263] Step 8:

[0264] The server notifies the user that a potentially fraudulent call was received and that the matter has been resolved. A summary of the conversation is reported if necessary.

[0265] (Example 1)

[0266] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0267] In light of the increasing prevalence of telephone-based fraud, this invention aims to effectively protect ordinary users, particularly the elderly, from becoming victims of fraud. It seeks to provide an environment where users can use telephone communication with peace of mind, without being directly affected by fraudulent calls. Furthermore, developing technology capable of responding quickly and accurately to increasingly diverse and sophisticated fraudulent activities is also a crucial objective.

[0268] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0269] In this invention, the server includes means for analyzing voice information received by a communication device and evaluating the probability of fraudulent activity, means for generating conversation content using natural language processing technology with a response generation device, and means for outputting the generated conversation content using speech synthesis technology. This makes it possible to detect fraudulent calls more quickly and accurately than with conventional manual responses, protect users from fraud, and ensure user safety.

[0270] "Communication equipment" refers to devices used to send and receive voice and data, and includes telephones, smartphones, and other similar devices.

[0271] "Voice information" refers to the voice data exchanged during a phone call, which is recorded as human voice.

[0272] "Analysis" refers to processing data and extracting information, and in particular, includes the process of converting audio data into text and evaluating it.

[0273] "Assessing the probability of fraud" is the process of numerically or qualitatively estimating the likelihood of fraud based on the information received.

[0274] A "response generation device" is a device that has the function of generating an appropriate response in natural language based on received audio information and analysis results.

[0275] "Natural language processing technology" refers to the technology that enables computers to understand, process, and generate human language.

[0276] "Speech synthesis technology" is a technology that outputs text information as speech, generating sound waveforms that closely resemble human voices.

[0277] "Dialogue information" refers to the content of conversations recorded during a phone call, which is stored for later analysis and learning.

[0278] "Storage device" refers to a device or system for storing data, and includes databases and cloud storage.

[0279] "Learning" refers to the process by which a machine learning model extracts knowledge and patterns using new and historical data, thereby improving its accuracy and performance.

[0280] A "database" is a collection of data that has a structure that organizes information and allows for efficient access.

[0281] "Immediately assessing the degree of suspicion" refers to the process of quickly determining the reliability of the received communication number and content, and immediately estimating the possibility of fraud.

[0282] The present invention involves deploying a system that combines communication equipment, servers, and user terminals to effectively protect users from fraudulent phone calls. It primarily implements functions such as voice information analysis, fraudulent activity evaluation, automated response generation, speech synthesis output, and fraud pattern learning.

[0283] The server utilizes speech recognition and natural language processing technologies to process audio information received through communication devices. Specifically, the server uses speech recognition software (e.g., a common speech recognition API) to convert audio into text and then evaluates the likelihood of fraudulent activity based on that text. This evaluation employs machine learning models to compare the received phone number with past suspicious cases registered in the database.

[0284] The user terminal receives an instruction from the server and automatically makes a response. The server uses a generative AI model to generate conversation content with natural language expressions based on a prompt sentence. As an example of a prompt sentence, content such as "Please generate a response to reassure the elderly about a fraud call" can be considered. The generated response is output as actual voice on the user terminal by voice synthesis technology. An existing voice synthesis engine is used for voice synthesis.

[0285] Such a system can safely handle the situation even when, for example, an elderly user becomes a target of phone fraud, without coming into contact with the fraudster. All conversation information is recorded on the server, and this data is used for the continuous learning of the generative AI model. As a result, the fraud detection ability of the entire system is improved, and it can quickly adapt to new fraud patterns.

[0286] The flow of the specific process in Example 1 will be described using FIG. 11.

[0287] Step 1:

[0288] When the user terminal receives an incoming call, it acquires the phone number and the information of the caller and transmits them to the server. This input includes the received phone number and voice data. The user terminal encrypts these data and transmits them to the server. This enables quick analysis while protecting personal information.

[0289] Step 2:

[0290] The server analyzes the received voice data. The input voice data is converted into text data using voice recognition technology. Next, the server checks the corresponding phone number against the database and evaluates whether it belongs to the suspicious list. This check immediately determines the degree of suspicion and evaluates the possibility of a fraud act.

[0291] Step 3:

[0292] If the server determines that a call is likely to be a scam, it sends a prompt to the AI ​​model. The prompt might say, "Generate a reassuring response for an elderly person in a scam call," and the AI ​​model generates a natural language dialogue. This generation process is optimized based on past data and similar cases.

[0293] Step 4:

[0294] The generated response is sent from the server to the user's terminal, which outputs it as actual voice using speech synthesis technology. This output uses speech synthesis that has a sound quality close to that of a human voice. The terminal continues to interact with the scammer, adjusting the volume and tone to ensure the conversation proceeds naturally.

[0295] Step 5:

[0296] The server records the content of conversations with scammers and stores it in a database. The recorded conversation information is used in subsequent learning processes. The server updates the generated AI model based on the stored data and continues to learn in order to detect new scam patterns. This improves the overall scam detection capability of the system.

[0297] (Application Example 1)

[0298] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0299] Traditional fraud prevention systems often only detect fraudulent activity using fixed methods, limiting their ability to combat the ever-evolving nature of fraud. Furthermore, these systems require frequent updates, leading to user-intensive operation and difficulties in responding immediately to new fraud techniques. Additionally, insufficient awareness campaigns about fraudulent calls targeting the elderly, coupled with the inability to respond in real-time, raise concerns about the potential for widespread victimization.

[0300] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0301] In this invention, the server includes means for analyzing voice data received by a communication device and determining the possibility of fraudulent activity, means for activating an automated response device and generating a response using natural language processing technology, and means for outputting the generated response using speech synthesis technology. This enables real-time detection and immediate response to fraudulent activity, ensuring safety for the elderly and other users. Users can receive immediate fraud warnings on their devices through a pre-installed mobile application. Furthermore, by linking with cloud services, it is possible to respond quickly to various fraud patterns, reduce the complexity of user operation, and flexibly respond to the latest fraudulent activities.

[0302] A "communication device" is a device that receives voice data, analyzes that data, and has the function of determining the possibility of fraudulent activity.

[0303] An "automated response device" is a device that can speak responses generated using natural language processing technology to fraudsters.

[0304] "Natural language processing technology" is a technology for analyzing, understanding, and generating human language, enabling smooth communication between machines and humans.

[0305] "Speech synthesis technology" is a technology that converts text data into speech data and outputs it as a natural-sounding human voice.

[0306] A "database" is a storage device for saving conversation data with scammers, allowing for quick retrieval and reference of data as needed.

[0307] A "mobile application" is software that runs on a user's mobile device to display the results of fraud detection and provide warnings.

[0308] A "cloud service" uses computing resources and data storage provided via the Internet and serves as a foundation for quickly transmitting and updating the discrimination results of fraud acts.

[0309] A "user interface" is an interface for a user to interact with a computer system and is designed for displaying and inputting information.

[0310] The implementation of this invention is mainly by a system in which three parties, namely a server, a terminal, and a user, cooperate to prevent fraud calls. As a specific form of the invention, it starts from the step of analyzing the voice data received by a communication device and discriminating the possibility of fraud acts. The server uses the communication device for this discrimination, collates the received phone number and voice data with a database through a cloud service, and evaluates the possibility of fraud.

[0311] Based on the discrimination result, if a fraud act is recognized, the server automatically activates an automatic response device. This device uses natural language processing technology to generate appropriate response content and speaks to the fraudster via voice synthesis technology. Thereby, while maintaining the conversation with the fraudster, data can be collected in a safe manner.

[0312] At the same time, a mobile application operates on the terminal. When a fraud act is discriminated, the result is transmitted through the cloud service, and a warning is given to the user via the user interface. Through this process, the user can quickly notice the fraud act and prevent damage.

[0313] As a concrete example, let's assume a user has this system installed on their smartphone and receives a call. The system immediately identifies the number "080-XXXX-XXXX" as a scam and responds with an automated voice message saying, "I'm sorry, but could you please tell me your name again?" During this time, the user receives a real-time warning on their mobile device and can take the necessary action.

[0314] Examples of prompts for a generative AI model include the following:

[0315] A scam call has been identified. Please ask the scammer the following question: "Excuse me, but could you please tell me your name again?"

[0316] This embodiment allows for flexible responses to the latest fraudulent methods and provides users with peace of mind.

[0317] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0318] Step 1:

[0319] The server obtains voice data and phone numbers received from the user's terminal via a communication device. The voice data, as input data, is subjected to a voice analysis algorithm, and the phone numbers are compared against a database in the cloud. This process calculates the likelihood of fraudulent activity.

[0320] Step 2:

[0321] The server determines the likelihood of fraud based on the phone number and the results of voice analysis. If it determines that there is a high probability of fraud, it retrieves the relevant information from the database and generates a trigger to activate the automated response system. The input in this step is the phone number and the analysis results, and the output is the fraud determination result.

[0322] Step 3:

[0323] The device receives the fraud detection result from the server and notifies the user through the mobile application. The application displays a warning message using the user interface and prompts the user to take action. The input is the detection result from the server, and the output is the warning display.

[0324] Step 4:

[0325] If the server determines that a user is a scammer, it uses an automated response system to generate a prompt message to respond to the scammer. This prompt message is generated using natural language processing technology and output in real time using speech synthesis technology. The input is the judgment result and the generating AI model, and the output is the synthesized response.

[0326] Step 5:

[0327] The server records the call data with the scammer and stores it in a database so that the AI ​​model can learn new scam patterns at a later date. In this step, the call data is used as input, and the output is the stored training data.

[0328] These processing steps enable the system to detect fraud in real time and notify users, providing a systematic approach to combating fraudulent activity.

[0329] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0330] The system of the present invention is comprised of a communication device, a server, a user terminal, and an emotion engine. The user terminal first detects an incoming call and transmits the corresponding voice data to the server and the emotion engine. The server captures the incoming number and voice data and determines the possibility of fraudulent activity by comparing it with a database. Based on this determination, it activates an automated response system and generates an appropriate response using natural language processing technology.

[0331] Furthermore, the emotion engine analyzes the voice data to recognize the user's emotional state. It extracts the type and intensity of the emotion and sends it to the server. The server then takes this emotional information into account and adjusts the response accordingly. For example, if the user is showing anxiety or confusion, the response can be changed to a gentler tone to provide reassurance.

[0332] The generated response is converted into speech using speech synthesis technology and spoken to the other party. The server records all resulting interactions with the scammer and stores them in a database as more detailed data. This allows the AI ​​model to use the emotional responses as training data for learning scam patterns.

[0333] For example, when a user receives a fraudulent phone call, the server checks the number and recognizes the possibility of fraud. If the emotion engine determines from the user's voice that they are remaining calm, the server generates a normal response accordingly, resulting in a natural conversation. On the other hand, if the emotion engine determines that the user is showing signs of anxiety, it adjusts the response to provide reassurance, such as, "It's okay, let's talk slowly."

[0334] This system enhances protection against fraudulent activities and ensures the emotional safety of users. Through these processes, users can be effectively protected from fraudulent phone calls.

[0335] The following describes the processing flow.

[0336] Step 1:

[0337] The device detects an incoming call. It sends the incoming call information and voice data to the server and the emotion engine.

[0338] Step 2:

[0339] The server compares the incoming number against a database of past fraudulent numbers to determine the likelihood of fraud. If the result indicates a high probability of fraud, the automated response system activation flag is set.

[0340] Step 3:

[0341] The emotion engine analyzes the received audio data and extracts the user's emotional state. It generates data about the type and intensity of the emotion and sends it to the server.

[0342] Step 4:

[0343] The server receives emotional data from the emotion engine and uses natural language processing techniques to adjust the response. For example, if the user is expressing anxiety, the response is modified to be more reassuring.

[0344] Step 5:

[0345] The server sends the generated response to the terminal. The terminal then uses speech synthesis technology to convert this response into speech.

[0346] Step 6:

[0347] The device speaks a voiced response to the scammer. The voice tone and tempo are set to mimic an elderly person's voice, reducing any unnaturalness.

[0348] Step 7:

[0349] The server continuously records conversations with scammers and stores all data in a database. This includes conversation content and user sentiment data.

[0350] Step 8:

[0351] Once the server determines that the conversation has ended, it uses the recorded data as training material for the AI ​​model. This improves its ability to respond to new fraud patterns and emotional responses.

[0352] Step 9:

[0353] The server notifies the user when the conversation has ended and, if necessary, reports a summary to confirm that the scam call was handled properly.

[0354] (Example 2)

[0355] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0356] Conventional fraud prevention systems rely on simple methods such as number matching to identify fraud, making it difficult to detect when fraudsters change their numbers. Furthermore, they lack systems that take into account the emotional reactions of users, which hinders their ability to enhance user confidence.

[0357] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0358] In this invention, the server includes means for analyzing audio data received by a communication device and determining the possibility of fraudulent activity; means for activating an automated response device and generating a response using natural language processing technology based on the determination result; and means for outputting the generated response using speech synthesis technology. This makes it possible to highly detect fraudulent activity through audio data, and at the same time enhance the user's sense of security and strengthen the defense against fraudulent activity by providing a response that takes into account the user's emotional state.

[0359] A "communication device" is a device that has the function of sending and receiving voice data, and usually has an interface for exchanging data with other devices via a network.

[0360] "Audio data" refers to data used to record and process audio in digital format, and includes the content of a call and its voice patterns.

[0361] "Fraud detection" is a process of evaluating the likelihood of fraud based on received data and detecting such activity.

[0362] An "automatic response device" is a device that autonomously generates a response through a program and provides that response to the user or other party in real time.

[0363] "Natural language processing technology" is a technology that understands, analyzes, and generates human language, making it possible for machines to communicate with humans in a natural way.

[0364] "Speech synthesis technology" is a technology that converts text data into speech data and is used to generate natural-sounding speech.

[0365] "Emotional state" refers to a person's psychological state inferred from their voice or text, and includes emotions such as anxiety, relief, and anger.

[0366] "Response content" refers to the content of natural-sounding conversations and messages generated based on the received data and emotional state.

[0367] A "database" is a structured data storage system for efficiently storing, managing, and accessing information.

[0368] A "generative AI model" is an algorithm that uses machine learning techniques to learn patterns from large amounts of data and then makes predictions or generates new data.

[0369] The present invention is implemented as a system combining a communication device, a computer server, a user terminal, and an emotion engine. This system aims to detect fraudulent calls and optimize user responses, and is realized through the use of various technologies.

[0370] First, when the user terminal detects an incoming call, it collects voice data in real time and transmits it to a computer server and emotion engine via the internet. The user terminal uses a secure protocol to ensure the safe transfer of data.

[0371] Next, the server uses a specialized algorithm to analyze the voice data and the incoming number. To determine the likelihood of fraud, it performs number verification in conjunction with a database. If a number registered as a fraudulent number is detected, the server activates an automated response system. Using natural language processing technology, it generates a response based on the received information, and then uses speech synthesis technology to convert that response into speech.

[0372] The emotion engine uses a voice analysis algorithm to extract and recognize the user's emotional state in real time from voice data. For example, if anxiety is detected from the voice tone or speed, that emotional information is sent to the server. The server then adjusts the response based on that emotional information to provide the user with a reassuring response.

[0373] For example, when a user receives a scam call, the server is configured to respond in a gentle tone, such as "Don't worry, we'll support you," if the emotion engine reports that the user is feeling anxious.

[0374] An example of a prompt message would be, "If a user receives a fraudulent phone call, please tell me how to respond if they express anxiety." This is used to instruct the generating AI model on specific actions to take.

[0375] This system consists of communication devices, terminals, and specialized software running on servers, and functions to prevent fraudulent activities and enhance user confidence. In this way, users can handle phone calls more safely and with greater peace of mind.

[0376] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0377] Step 1:

[0378] Incoming call detection and data collection

[0379] The terminal detects incoming calls. Using a communication device API, it acquires the incoming number and voice data in real time. Inputs include the incoming call event, phone number, and voice signal, while output is the acquired voice data, prepared for transmission to the server and emotion engine.

[0380] Step 2:

[0381] Sending audio data

[0382] The terminal transmits the received voice data to the server and emotion engine via a secure communication protocol (e.g., HTTPS). Inputs include voice data and the incoming number, while output is the transmission of this data to the server and emotion engine.

[0383] Step 3:

[0384] Identifying fraudulent activity

[0385] The server uses the received voice data and incoming number to determine the possibility of fraud. It accesses a database and compares the incoming number against a list of known fraudulent numbers. The input is the incoming number and voice data, and the output is either a flag indicating the possibility of fraud or a status of "not found".

[0386] Step 4:

[0387] Analysis of emotional states

[0388] The emotion engine analyzes voice data to determine the user's emotional state. It analyzes voice tone, pitch, speed, etc., to extract emotional information. The input is voice data, and the output is the type of emotion (e.g., anxiety, relief) and its intensity.

[0389] Step 5:

[0390] Generating the response content

[0391] The server considers the presence or absence of fraudulent activity and sentiment information, and uses a generative AI model to create an appropriate response. The prompt in this case is in the form of, "If the user is showing anxiety, please generate an appropriate response." The input consists of a discrimination flag and sentiment data, and the output is the generated text of an optimized response.

[0392] Step 6:

[0393] Speech synthesis and responsive speech

[0394] The server uses speech synthesis technology to convert the generated response into speech and automatically plays it during the call with the scammer. The input is the generated text data, and the output is the generated audio data that is spoken to the other party on the call.

[0395] Step 7:

[0396] Data recording and storage

[0397] The server records and stores data in a database, including all interactions related to fraudulent activity and the emotional responses at the time. Inputs include conversation log data and emotional information, while output is this information stored in a structured format, making it available as subsequent training data.

[0398] (Application Example 2)

[0399] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0400] Conventional fraud prevention systems focus on identifying fraudulent activity through the analysis of voice data, but they have the drawback of lacking consideration for the user's emotional state. In particular, when a user experiences anxiety or confusion due to a fraudulent call, the system may not be able to respond appropriately, potentially compromising the user's emotional safety.

[0401] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0402] In this invention, the server includes means for analyzing audio information received by communication equipment and determining the possibility of fraudulent activity; means for analyzing the emotional state from the received audio information and adjusting the response content based on the user's emotions; and means for recording the conversation record with the fraudster in a storage device and utilizing it as a source of information for learning new fraudulent activity patterns. This makes it possible to strengthen the defense against fraudulent activity and ensure the emotional safety of the user.

[0403] "Communication equipment" is a general term for devices and terminals used to receive and analyze voice information.

[0404] "Audio information" refers to data captured by communication devices as audio signals, and is used as material for analysis and response generation.

[0405] A "discrimination method" is a system for evaluating and judging the possibility of fraudulent activity based on audio information.

[0406] An "automatic response device" is a system that generates an appropriate response based on the discrimination result.

[0407] "Natural language processing technology" is a technology that allows computers to understand and generate human language, and is used to generate response content.

[0408] "Emotional state" refers to the type and intensity of the user's emotions, as analyzed from their voice information.

[0409] "Response content" refers to the response to the user generated based on the analysis of the voice information.

[0410] "Speech synthesis technology" is a technology that converts generated responses into speech and outputs them.

[0411] A "conversation record" is data that records and saves the content of conversations with a scammer.

[0412] A "storage device" is a medium for storing conversation records and using them for later learning and analysis.

[0413] "Information sources" refer to data that provides the necessary information for learning about new fraudulent activities, and include conversation records, etc.

[0414] The system for carrying out this invention consists of a communication device, a server, an emotion recognition engine, an automatic response device, and a speech synthesis device. The communication device is responsible for receiving voice information from the user and transmitting it to the server. The server analyzes the received voice information and first determines the possibility of fraudulent activity. This determination is made by comparing the contact number and voice data with a database of past fraudulent activities. Furthermore, the server uses the emotion recognition engine to analyze the user's emotional state from the voice information. The emotion recognition engine evaluates the user's emotions based on parameters such as voice waveform, frequency, and speed.

[0415] The server then generates the optimal response that the automated response system should provide to the user, based on the likelihood of fraud and the user's emotional state. Utilizing natural language processing technology, the response is tailored to soothe the user's emotions. The generated response is then converted into natural-sounding speech by a speech synthesizer and returned to the user via communication equipment.

[0416] Furthermore, all conversation records are recorded and stored in a storage device. The stored data is used as information for the server to learn new fraud patterns. This allows the system to continuously improve the accuracy of fraud detection.

[0417] For example, if a user receives a suspicious call, the server immediately analyzes the audio information and, if it determines that there is a high probability of fraud, it retrieves data indicating that the user's emotions are unstable. In response, a reassuring response such as "It's okay, let's talk slowly?" is generated. An example of a prompt to the generating AI model would be, "Analyze the audio data of this call and assess the possibility of a fraudulent transaction. If the user is feeling anxious, generate an appropriate support message."

[0418] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0419] Step 1:

[0420] Communication equipment receives voice information from a user. The input is the user's spoken voice, and the output is digitized voice data. Voice sampling is performed to convert the voice into a digital signal.

[0421] Step 2:

[0422] The server receives the audio data and analyzes it to determine the possibility of fraudulent activity. The input is the audio data from step 1, and the output is the evaluation result of the fraud possibility. The server refers to the database, matches the audio data and contact numbers, and performs fraud pattern matching.

[0423] Step 3:

[0424] The server activates an emotion recognition engine and analyzes the user's emotional state from the audio data. The input is audio data, and the output is the user's emotion evaluation result. Audio parameters (waveform, audio frequency, speed, etc.) are calculated to identify the emotion.

[0425] Step 4:

[0426] The server applies natural language processing techniques to generate the optimal response based on the discrimination result and emotional state. The input is the fraud discrimination result and the emotional evaluation result, and the output is the generated text response. A generative AI model performs the language generation process.

[0427] Step 5:

[0428] A speech synthesizer converts the generated text response into speech. The input is the generated text response, and the output is the synthesized speech. A speech synthesis engine converts text into speech.

[0429] Step 6:

[0430] The server generates audio and returns it to the user via the communication device. The input is synthesized speech, and the output is the audio provided to the user. The communication device plays the audio.

[0431] Step 7:

[0432] The server saves conversation records and stores them in a database. Inputs are log information and audio recordings from all processes, and output is the conversation data stored in the database. A storage solution is used for data storage.

[0433] Step 8:

[0434] The server learns new fraud patterns using stored data. The input is conversation records in the database, and the output is an updated fraud detection model. A machine learning algorithm performs the learning process.

[0435] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0436] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0437] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0438] [Third Embodiment]

[0439] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0440] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0441] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0442] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0443] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0444] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0445] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0446] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0447] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0448] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0449] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0450] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0451] The fraudulent call prevention system of the present invention consists of a communication device, a server, a user terminal, and a program that links these components. The user terminal receives an incoming call, performs initial processing, and sends the information to the server. The server analyzes the received phone number and voice data to determine the possibility of fraudulent activity.

[0452] If fraudulent activity is detected, the server, through an automated response system, speaks a conversation generated using natural language processing technology to the fraudster. The terminal then uses speech synthesis technology to convert this generated response into speech and processes it to allow the call with the fraudster to continue.

[0453] For example, if a user's device receives a call and the number is on a past list of suspicious numbers, the server immediately determines that it is highly likely to be a scam. The server activates an automated answering system and generates a natural-sounding response, such as, "I can't hear you very well, could you please repeat your name?" This synthesized response is spoken to the scammer in real time.

[0454] All interactions with scammers are recorded on the server, and an AI model later reviews this data to learn new scam patterns. This system ensures that users are safely protected from the direct impact of scam calls. Furthermore, the recorded data contributes to overall system improvements and enhances the system's ability to detect scams more effectively.

[0455] By implementing this invention, it is possible to significantly reduce the damage caused by telephone scams targeting the elderly, and it is expected that this will improve social safety.

[0456] The following describes the processing flow.

[0457] Step 1:

[0458] The device detects an incoming call. It sends a message to the server, along with audio data, indicating that an incoming call has been received.

[0459] Step 2:

[0460] The server receives the incoming phone number and voice data and determines whether it is likely to be a scam. By comparing it with a stored database of past scams, it determines whether or not it is likely to be a scam.

[0461] Step 3:

[0462] Based on the server's assessment, if it determines there is a high probability of fraud, it activates an automated response system. Natural language processing technology is used to generate responses that mimic those of an elderly person.

[0463] Step 4:

[0464] The server sends the generated response to the terminal. This response might be something like, "Could you please repeat that slowly?"

[0465] Step 5:

[0466] The terminal receives a response from the server, converts it into speech using speech synthesis technology, and speaks it to the other party. To ensure the speech doesn't sound unnatural, it maintains a natural tone, mimicking that of an elderly person.

[0467] Step 6:

[0468] The server records all conversations with the scammer. Repeat steps 3 through 5 as long as the conversation continues.

[0469] Step 7:

[0470] Once the conversation ends, the server saves the recorded data to a database. This data is used to train AI models, improving their accuracy in responding to new fraudulent schemes.

[0471] Step 8:

[0472] The server notifies the user that a potentially fraudulent call was received and that the matter has been resolved. A summary of the conversation is reported if necessary.

[0473] (Example 1)

[0474] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0475] In light of the increasing prevalence of telephone-based fraud, this invention aims to effectively protect ordinary users, particularly the elderly, from becoming victims of fraud. It seeks to provide an environment where users can use telephone communication with peace of mind, without being directly affected by fraudulent calls. Furthermore, developing technology capable of responding quickly and accurately to increasingly diverse and sophisticated fraudulent activities is also a crucial objective.

[0476] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0477] In this invention, the server includes means for analyzing voice information received by a communication device and evaluating the probability of fraudulent activity, means for generating conversation content using natural language processing technology with a response generation device, and means for outputting the generated conversation content using speech synthesis technology. This makes it possible to detect fraudulent calls more quickly and accurately than with conventional manual responses, protect users from fraud, and ensure user safety.

[0478] "Communication equipment" refers to devices used to send and receive voice and data, and includes telephones, smartphones, and other similar devices.

[0479] "Voice information" refers to the voice data exchanged during a phone call, which is recorded as human voice.

[0480] "Analysis" refers to processing data and extracting information, and in particular, includes the process of converting audio data into text and evaluating it.

[0481] "Assessing the probability of fraud" is the process of numerically or qualitatively estimating the likelihood of fraud based on the information received.

[0482] A "response generation device" is a device that has the function of generating an appropriate response in natural language based on received audio information and analysis results.

[0483] "Natural language processing technology" refers to the technology that enables computers to understand, process, and generate human language.

[0484] "Speech synthesis technology" is a technology that outputs text information as speech, generating sound waveforms that closely resemble human voices.

[0485] "Dialogue information" refers to the content of conversations recorded during a phone call, which is stored for later analysis and learning.

[0486] "Storage device" refers to a device or system for storing data, and includes databases and cloud storage.

[0487] "Learning" refers to the process by which a machine learning model extracts knowledge and patterns using new and historical data, thereby improving its accuracy and performance.

[0488] A "database" is a collection of data that has a structure that organizes information and allows for efficient access.

[0489] "Immediately assessing the degree of suspicion" refers to the process of quickly determining the reliability of the received communication number and content, and immediately estimating the possibility of fraud.

[0490] The present invention involves deploying a system that combines communication equipment, servers, and user terminals to effectively protect users from fraudulent phone calls. It primarily implements functions such as voice information analysis, fraudulent activity evaluation, automated response generation, speech synthesis output, and fraud pattern learning.

[0491] The server utilizes speech recognition and natural language processing technologies to process audio information received through communication devices. Specifically, the server uses speech recognition software (e.g., a common speech recognition API) to convert audio into text and then evaluates the likelihood of fraudulent activity based on that text. This evaluation employs machine learning models to compare the received phone number with past suspicious cases registered in the database.

[0492] The user terminal receives instructions from the server and automatically responds. The server uses a generative AI model to generate natural-sounding dialogue based on the prompt text. An example of a prompt text might be, "Generate a response that will reassure an elderly person in a fraudulent phone call." This generated response is output as actual voice on the user terminal using speech synthesis technology. An existing speech synthesis engine is used for speech synthesis.

[0493] Such a system allows, for example, elderly users who become targets of phone scams to deal with the situation safely without having to come into contact with the scammers. All conversational information is recorded on a server, and this data is used for the continuous learning of the generating AI model. This improves the overall fraud detection capability of the system and allows it to quickly adapt to new fraud patterns.

[0494] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0495] Step 1:

[0496] When a user terminal receives an incoming call, it obtains the phone number and caller information and sends it to the server. This input includes the received phone number and voice data. The user terminal encrypts this data and sends it to the server. This ensures the protection of personal information while enabling rapid analysis.

[0497] Step 2:

[0498] The server analyzes the received audio data. The input audio data is converted into text data using speech recognition technology. Next, the server compares the phone number with the database and evaluates whether it falls under the suspicious list. This comparison allows for an immediate determination of the degree of suspicion and an assessment of the possibility of fraudulent activity.

[0499] Step 3:

[0500] If the server determines that a call is likely to be a scam, it sends a prompt to the AI ​​model. The prompt might say, "Generate a reassuring response for an elderly person in a scam call," and the AI ​​model generates a natural language dialogue. This generation process is optimized based on past data and similar cases.

[0501] Step 4:

[0502] The generated response is sent from the server to the user's terminal, which outputs it as actual voice using speech synthesis technology. This output uses speech synthesis that has a sound quality close to that of a human voice. The terminal continues to interact with the scammer, adjusting the volume and tone to ensure the conversation proceeds naturally.

[0503] Step 5:

[0504] The server records the content of conversations with scammers and stores it in a database. The recorded conversation information is used in subsequent learning processes. The server updates the generated AI model based on the stored data and continues to learn in order to detect new scam patterns. This improves the overall scam detection capability of the system.

[0505] (Application Example 1)

[0506] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0507] Traditional fraud prevention systems often only detect fraudulent activity using fixed methods, limiting their ability to combat the ever-evolving nature of fraud. Furthermore, these systems require frequent updates, leading to user-intensive operation and difficulties in responding immediately to new fraud techniques. Additionally, insufficient awareness campaigns about fraudulent calls targeting the elderly, coupled with the inability to respond in real-time, raise concerns about the potential for widespread victimization.

[0508] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0509] In this invention, the server includes means for analyzing voice data received by a communication device and determining the possibility of fraudulent activity, means for activating an automated response device and generating a response using natural language processing technology, and means for outputting the generated response using speech synthesis technology. This enables real-time detection and immediate response to fraudulent activity, ensuring safety for the elderly and other users. Users can receive immediate fraud warnings on their devices through a pre-installed mobile application. Furthermore, by linking with cloud services, it is possible to respond quickly to various fraud patterns, reduce the complexity of user operation, and flexibly respond to the latest fraudulent activities.

[0510] A "communication device" is a device that receives voice data, analyzes that data, and has the function of determining the possibility of fraudulent activity.

[0511] An "automated response device" is a device that can speak responses generated using natural language processing technology to fraudsters.

[0512] "Natural language processing technology" is a technology for analyzing, understanding, and generating human language, enabling smooth communication between machines and humans.

[0513] "Speech synthesis technology" is a technology that converts text data into speech data and outputs it as a natural-sounding human voice.

[0514] A "database" is a storage device for saving conversation data with scammers, allowing for quick retrieval and reference of data as needed.

[0515] A "mobile application" is software that runs on a user's mobile device to display the results of fraud detection and provide warnings.

[0516] "Cloud services" utilize computing resources and data storage provided via the internet, and serve as a foundation for rapidly transmitting and updating the results of fraud detection.

[0517] A "user interface" is an interface that allows a user to interact with a computer system, and is designed for displaying and inputting information.

[0518] The implementation of this invention primarily involves a system in which a server, a terminal, and a user cooperate to prevent fraudulent phone calls. In its specific form, the invention begins with a step in which a communication device analyzes received voice data to determine the possibility of fraudulent activity. The server utilizes a communication device for this determination, comparing the received phone number and voice data with a database via a cloud service to evaluate the likelihood of fraud.

[0519] Based on the detection results, if fraudulent activity is detected, the server automatically activates an automated response system. This system uses natural language processing technology to generate an appropriate response and speaks it to the fraudster via speech synthesis technology. This allows for secure data collection while maintaining a call with the fraudster.

[0520] Simultaneously, a mobile application runs on the device, and when fraudulent activity is detected, the results are sent via a cloud service, and the user is warned through the user interface. This process allows users to quickly recognize fraudulent activity and prevent becoming a victim.

[0521] As a concrete example, let's assume a user has this system installed on their smartphone and receives a call. The system immediately identifies the number "080-XXXX-XXXX" as a scam and responds with an automated voice message saying, "I'm sorry, but could you please tell me your name again?" During this time, the user receives a real-time warning on their mobile device and can take the necessary action.

[0522] Examples of prompts for a generative AI model include the following:

[0523] A scam call has been identified. Please ask the scammer the following question: "Excuse me, but could you please tell me your name again?"

[0524] This embodiment allows for flexible responses to the latest fraudulent methods and provides users with peace of mind.

[0525] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0526] Step 1:

[0527] The server obtains voice data and phone numbers received from the user's terminal via a communication device. The voice data, as input data, is subjected to a voice analysis algorithm, and the phone numbers are compared against a database in the cloud. This process calculates the likelihood of fraudulent activity.

[0528] Step 2:

[0529] The server determines the likelihood of fraud based on the phone number and the results of voice analysis. If it determines that there is a high probability of fraud, it retrieves the relevant information from the database and generates a trigger to activate the automated response system. The input in this step is the phone number and the analysis results, and the output is the fraud determination result.

[0530] Step 3:

[0531] The device receives the fraud detection result from the server and notifies the user through the mobile application. The application displays a warning message using the user interface and prompts the user to take action. The input is the detection result from the server, and the output is the warning display.

[0532] Step 4:

[0533] If the server determines that a user is a scammer, it uses an automated response system to generate a prompt message to respond to the scammer. This prompt message is generated using natural language processing technology and output in real time using speech synthesis technology. The input is the judgment result and the generating AI model, and the output is the synthesized response.

[0534] Step 5:

[0535] The server records the call data with the scammer and stores it in a database so that the AI ​​model can learn new scam patterns at a later date. In this step, the call data is used as input, and the output is the stored training data.

[0536] These processing steps enable the system to detect fraud in real time and notify users, providing a systematic approach to combating fraudulent activity.

[0537] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0538] The system of the present invention is comprised of a communication device, a server, a user terminal, and an emotion engine. The user terminal first detects an incoming call and transmits the corresponding voice data to the server and the emotion engine. The server captures the incoming number and voice data and determines the possibility of fraudulent activity by comparing it with a database. Based on this determination, it activates an automated response system and generates an appropriate response using natural language processing technology.

[0539] Furthermore, the emotion engine analyzes the voice data to recognize the user's emotional state. It extracts the type and intensity of the emotion and sends it to the server. The server then takes this emotional information into account and adjusts the response accordingly. For example, if the user is showing anxiety or confusion, the response can be changed to a gentler tone to provide reassurance.

[0540] The generated response is converted into speech using speech synthesis technology and spoken to the other party. The server records all resulting interactions with the scammer and stores them in a database as more detailed data. This allows the AI ​​model to use the emotional responses as training data for learning scam patterns.

[0541] For example, when a user receives a fraudulent phone call, the server checks the number and recognizes the possibility of fraud. If the emotion engine determines from the user's voice that they are remaining calm, the server generates a normal response accordingly, resulting in a natural conversation. On the other hand, if the emotion engine determines that the user is showing signs of anxiety, it adjusts the response to provide reassurance, such as, "It's okay, let's talk slowly."

[0542] This system enhances protection against fraudulent activities and ensures the emotional safety of users. Through these processes, users can be effectively protected from fraudulent phone calls.

[0543] The following describes the processing flow.

[0544] Step 1:

[0545] The device detects an incoming call. It sends the incoming call information and voice data to the server and the emotion engine.

[0546] Step 2:

[0547] The server compares the incoming number against a database of past fraudulent numbers to determine the likelihood of fraud. If the result indicates a high probability of fraud, the automated response system activation flag is set.

[0548] Step 3:

[0549] The emotion engine analyzes the received audio data and extracts the user's emotional state. It generates data about the type and intensity of the emotion and sends it to the server.

[0550] Step 4:

[0551] The server receives emotional data from the emotion engine and uses natural language processing techniques to adjust the response. For example, if the user is expressing anxiety, the response is modified to be more reassuring.

[0552] Step 5:

[0553] The server sends the generated response to the terminal. The terminal then uses speech synthesis technology to convert this response into speech.

[0554] Step 6:

[0555] The device speaks a voiced response to the scammer. The voice tone and tempo are set to mimic an elderly person's voice, reducing any unnaturalness.

[0556] Step 7:

[0557] The server continuously records conversations with scammers and stores all data in a database. This includes conversation content and user sentiment data.

[0558] Step 8:

[0559] Once the server determines that the conversation has ended, it uses the recorded data as training material for the AI ​​model. This improves its ability to respond to new fraud patterns and emotional responses.

[0560] Step 9:

[0561] The server notifies the user when the conversation has ended and, if necessary, reports a summary to confirm that the scam call was handled properly.

[0562] (Example 2)

[0563] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0564] Conventional fraud prevention systems rely on simple methods such as number matching to identify fraud, making it difficult to detect when fraudsters change their numbers. Furthermore, they lack systems that take into account the emotional reactions of users, which hinders their ability to enhance user confidence.

[0565] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0566] In this invention, the server includes means for analyzing audio data received by a communication device and determining the possibility of fraudulent activity; means for activating an automated response device and generating a response using natural language processing technology based on the determination result; and means for outputting the generated response using speech synthesis technology. This makes it possible to highly detect fraudulent activity through audio data, and at the same time enhance the user's sense of security and strengthen the defense against fraudulent activity by providing a response that takes into account the user's emotional state.

[0567] A "communication device" is a device that has the function of sending and receiving voice data, and usually has an interface for exchanging data with other devices via a network.

[0568] "Audio data" refers to data used to record and process audio in digital format, and includes the content of a call and its voice patterns.

[0569] "Fraud detection" is a process of evaluating the likelihood of fraud based on received data and detecting such activity.

[0570] An "automatic response device" is a device that autonomously generates a response through a program and provides that response to the user or other party in real time.

[0571] "Natural language processing technology" is a technology that understands, analyzes, and generates human language, making it possible for machines to communicate with humans in a natural way.

[0572] "Speech synthesis technology" is a technology that converts text data into speech data and is used to generate natural-sounding speech.

[0573] "Emotional state" refers to a person's psychological state inferred from their voice or text, and includes emotions such as anxiety, relief, and anger.

[0574] "Response content" refers to the content of natural-sounding conversations and messages generated based on the received data and emotional state.

[0575] A "database" is a structured data storage system for efficiently storing, managing, and accessing information.

[0576] A "generative AI model" is an algorithm that uses machine learning techniques to learn patterns from large amounts of data and then makes predictions or generates new data.

[0577] The present invention is implemented as a system combining a communication device, a computer server, a user terminal, and an emotion engine. This system aims to detect fraudulent calls and optimize user responses, and is realized through the use of various technologies.

[0578] First, when the user terminal detects an incoming call, it collects voice data in real time and transmits it to a computer server and emotion engine via the internet. The user terminal uses a secure protocol to ensure the safe transfer of data.

[0579] Next, the server uses a specialized algorithm to analyze the voice data and the incoming number. To determine the likelihood of fraud, it performs number verification in conjunction with a database. If a number registered as a fraudulent number is detected, the server activates an automated response system. Using natural language processing technology, it generates a response based on the received information, and then uses speech synthesis technology to convert that response into speech.

[0580] The emotion engine uses a voice analysis algorithm to extract and recognize the user's emotional state in real time from voice data. For example, if anxiety is detected from the voice tone or speed, that emotional information is sent to the server. The server then adjusts the response based on that emotional information to provide the user with a reassuring response.

[0581] For example, when a user receives a scam call, the server is configured to respond in a gentle tone, such as "Don't worry, we'll support you," if the emotion engine reports that the user is feeling anxious.

[0582] An example of a prompt message would be, "If a user receives a fraudulent phone call, please tell me how to respond if they express anxiety." This is used to instruct the generating AI model on specific actions to take.

[0583] This system consists of communication devices, terminals, and specialized software running on servers, and functions to prevent fraudulent activities and enhance user confidence. In this way, users can handle phone calls more safely and with greater peace of mind.

[0584] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0585] Step 1:

[0586] Incoming call detection and data collection

[0587] The terminal detects incoming calls. Using a communication device API, it acquires the incoming number and voice data in real time. Inputs include the incoming call event, phone number, and voice signal, while output is the acquired voice data, prepared for transmission to the server and emotion engine.

[0588] Step 2:

[0589] Sending audio data

[0590] The terminal transmits the received voice data to the server and emotion engine via a secure communication protocol (e.g., HTTPS). Inputs include voice data and the incoming number, while output is the transmission of this data to the server and emotion engine.

[0591] Step 3:

[0592] Identifying fraudulent activity

[0593] The server uses the received voice data and incoming number to determine the possibility of fraud. It accesses a database and compares the incoming number against a list of known fraudulent numbers. The input is the incoming number and voice data, and the output is either a flag indicating the possibility of fraud or a status of "not found".

[0594] Step 4:

[0595] Analysis of emotional states

[0596] The emotion engine analyzes voice data to determine the user's emotional state. It analyzes voice tone, pitch, speed, etc., to extract emotional information. The input is voice data, and the output is the type of emotion (e.g., anxiety, relief) and its intensity.

[0597] Step 5:

[0598] Generating the response content

[0599] The server considers the presence or absence of fraudulent activity and sentiment information, and uses a generative AI model to create an appropriate response. The prompt in this case is in the form of, "If the user is showing anxiety, please generate an appropriate response." The input consists of a discrimination flag and sentiment data, and the output is the generated text of an optimized response.

[0600] Step 6:

[0601] Speech synthesis and responsive speech

[0602] The server uses speech synthesis technology to convert the generated response into speech and automatically plays it during the call with the scammer. The input is the generated text data, and the output is the generated audio data that is spoken to the other party on the call.

[0603] Step 7:

[0604] Data recording and storage

[0605] The server records and stores data in a database, including all interactions related to fraudulent activity and the emotional responses at the time. Inputs include conversation log data and emotional information, while output is this information stored in a structured format, making it available as subsequent training data.

[0606] (Application Example 2)

[0607] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0608] Conventional fraud prevention systems focus on identifying fraudulent activity through the analysis of voice data, but they have the drawback of lacking consideration for the user's emotional state. In particular, when a user experiences anxiety or confusion due to a fraudulent call, the system may not be able to respond appropriately, potentially compromising the user's emotional safety.

[0609] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0610] In this invention, the server includes means for analyzing audio information received by communication equipment and determining the possibility of fraudulent activity; means for analyzing the emotional state from the received audio information and adjusting the response content based on the user's emotions; and means for recording the conversation record with the fraudster in a storage device and utilizing it as a source of information for learning new fraudulent activity patterns. This makes it possible to strengthen the defense against fraudulent activity and ensure the emotional safety of the user.

[0611] "Communication equipment" is a general term for devices and terminals used to receive and analyze voice information.

[0612] "Audio information" refers to data captured by communication devices as audio signals, and is used as material for analysis and response generation.

[0613] A "discrimination method" is a system for evaluating and judging the possibility of fraudulent activity based on audio information.

[0614] An "automatic response device" is a system that generates an appropriate response based on the discrimination result.

[0615] "Natural language processing technology" is a technology that allows computers to understand and generate human language, and is used to generate response content.

[0616] "Emotional state" refers to the type and intensity of the user's emotions, as analyzed from their voice information.

[0617] "Response content" refers to the response to the user generated based on the analysis of the voice information.

[0618] "Speech synthesis technology" is a technology that converts generated responses into speech and outputs them.

[0619] A "conversation record" is data that records and saves the content of conversations with a scammer.

[0620] A "storage device" is a medium for storing conversation records and using them for later learning and analysis.

[0621] "Information sources" refer to data that provides the necessary information for learning about new fraudulent activities, and include conversation records, etc.

[0622] The system for carrying out this invention consists of a communication device, a server, an emotion recognition engine, an automatic response device, and a speech synthesis device. The communication device is responsible for receiving voice information from the user and transmitting it to the server. The server analyzes the received voice information and first determines the possibility of fraudulent activity. This determination is made by comparing the contact number and voice data with a database of past fraudulent activities. Furthermore, the server uses the emotion recognition engine to analyze the user's emotional state from the voice information. The emotion recognition engine evaluates the user's emotions based on parameters such as voice waveform, frequency, and speed.

[0623] The server then generates the optimal response that the automated response system should provide to the user, based on the likelihood of fraud and the user's emotional state. Utilizing natural language processing technology, the response is tailored to soothe the user's emotions. The generated response is then converted into natural-sounding speech by a speech synthesizer and returned to the user via communication equipment.

[0624] Furthermore, all conversation records are recorded and stored in a storage device. The stored data is used as information for the server to learn new fraud patterns. This allows the system to continuously improve the accuracy of fraud detection.

[0625] For example, if a user receives a suspicious call, the server immediately analyzes the audio information and, if it determines that there is a high probability of fraud, it retrieves data indicating that the user's emotions are unstable. In response, a reassuring response such as "It's okay, let's talk slowly?" is generated. An example of a prompt to the generating AI model would be, "Analyze the audio data of this call and assess the possibility of a fraudulent transaction. If the user is feeling anxious, generate an appropriate support message."

[0626] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0627] Step 1:

[0628] Communication equipment receives voice information from a user. The input is the user's spoken voice, and the output is digitized voice data. Voice sampling is performed to convert the voice into a digital signal.

[0629] Step 2:

[0630] The server receives the audio data and analyzes it to determine the possibility of fraudulent activity. The input is the audio data from step 1, and the output is the evaluation result of the fraud possibility. The server refers to the database, matches the audio data and contact numbers, and performs fraud pattern matching.

[0631] Step 3:

[0632] The server activates an emotion recognition engine and analyzes the user's emotional state from the audio data. The input is audio data, and the output is the user's emotion evaluation result. Audio parameters (waveform, audio frequency, speed, etc.) are calculated to identify the emotion.

[0633] Step 4:

[0634] The server applies natural language processing techniques to generate the optimal response based on the discrimination result and emotional state. The input is the fraud discrimination result and the emotional evaluation result, and the output is the generated text response. A generative AI model performs the language generation process.

[0635] Step 5:

[0636] A speech synthesizer converts the generated text response into speech. The input is the generated text response, and the output is the synthesized speech. A speech synthesis engine converts text into speech.

[0637] Step 6:

[0638] The server generates audio and returns it to the user via the communication device. The input is synthesized speech, and the output is the audio provided to the user. The communication device plays the audio.

[0639] Step 7:

[0640] The server saves conversation records and stores them in a database. Inputs are log information and audio recordings from all processes, and output is the conversation data stored in the database. A storage solution is used for data storage.

[0641] Step 8:

[0642] The server learns new fraud patterns using stored data. The input is conversation records in the database, and the output is an updated fraud detection model. A machine learning algorithm performs the learning process.

[0643] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0644] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0645] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0646] [Fourth Embodiment]

[0647] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0648] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0649] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0650] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0651] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0652] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0653] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0654] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0655] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0656] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0657] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0658] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0659] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0660] The fraudulent call prevention system of the present invention consists of a communication device, a server, a user terminal, and a program that links these components. The user terminal receives an incoming call, performs initial processing, and sends the information to the server. The server analyzes the received phone number and voice data to determine the possibility of fraudulent activity.

[0661] If fraudulent activity is detected, the server, through an automated response system, speaks a conversation generated using natural language processing technology to the fraudster. The terminal then uses speech synthesis technology to convert this generated response into speech and processes it to allow the call with the fraudster to continue.

[0662] For example, if a user's device receives a call and the number is on a past list of suspicious numbers, the server immediately determines that it is highly likely to be a scam. The server activates an automated answering system and generates a natural-sounding response, such as, "I can't hear you very well, could you please repeat your name?" This synthesized response is spoken to the scammer in real time.

[0663] All interactions with scammers are recorded on the server, and an AI model later reviews this data to learn new scam patterns. This system ensures that users are safely protected from the direct impact of scam calls. Furthermore, the recorded data contributes to overall system improvements and enhances the system's ability to detect scams more effectively.

[0664] By implementing this invention, it is possible to significantly reduce the damage caused by telephone scams targeting the elderly, and it is expected that this will improve social safety.

[0665] The following describes the processing flow.

[0666] Step 1:

[0667] The device detects an incoming call. It sends a message to the server, along with audio data, indicating that an incoming call has been received.

[0668] Step 2:

[0669] The server receives the incoming phone number and voice data and determines whether it is likely to be a scam. By comparing it with a stored database of past scams, it determines whether or not it is likely to be a scam.

[0670] Step 3:

[0671] Based on the server's assessment, if it determines there is a high probability of fraud, it activates an automated response system. Natural language processing technology is used to generate responses that mimic those of an elderly person.

[0672] Step 4:

[0673] The server sends the generated response to the terminal. This response might be something like, "Could you please repeat that slowly?"

[0674] Step 5:

[0675] The terminal receives a response from the server, converts it into speech using speech synthesis technology, and speaks it to the other party. To ensure the speech doesn't sound unnatural, it maintains a natural tone, mimicking that of an elderly person.

[0676] Step 6:

[0677] The server records all conversations with the scammer. Repeat steps 3 through 5 as long as the conversation continues.

[0678] Step 7:

[0679] Once the conversation ends, the server saves the recorded data to a database. This data is used to train AI models, improving their accuracy in responding to new fraudulent schemes.

[0680] Step 8:

[0681] The server notifies the user that a potentially fraudulent call was received and that the matter has been resolved. A summary of the conversation is reported if necessary.

[0682] (Example 1)

[0683] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0684] In light of the increasing prevalence of telephone-based fraud, this invention aims to effectively protect ordinary users, particularly the elderly, from becoming victims of fraud. It seeks to provide an environment where users can use telephone communication with peace of mind, without being directly affected by fraudulent calls. Furthermore, developing technology capable of responding quickly and accurately to increasingly diverse and sophisticated fraudulent activities is also a crucial objective.

[0685] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0686] In this invention, the server includes means for analyzing voice information received by a communication device and evaluating the probability of fraudulent activity, means for generating conversation content using natural language processing technology with a response generation device, and means for outputting the generated conversation content using speech synthesis technology. This makes it possible to detect fraudulent calls more quickly and accurately than with conventional manual responses, protect users from fraud, and ensure user safety.

[0687] "Communication equipment" refers to devices used to send and receive voice and data, and includes telephones, smartphones, and other similar devices.

[0688] "Voice information" refers to the voice data exchanged during a phone call, which is recorded as human voice.

[0689] "Analysis" refers to processing data and extracting information, and in particular, includes the process of converting audio data into text and evaluating it.

[0690] "Assessing the probability of fraud" is the process of numerically or qualitatively estimating the likelihood of fraud based on the information received.

[0691] A "response generation device" is a device that has the function of generating an appropriate response in natural language based on received audio information and analysis results.

[0692] "Natural language processing technology" refers to the technology that enables computers to understand, process, and generate human language.

[0693] "Speech synthesis technology" is a technology that outputs text information as speech, generating sound waveforms that closely resemble human voices.

[0694] "Dialogue information" refers to the content of conversations recorded during a phone call, which is stored for later analysis and learning.

[0695] "Storage device" refers to a device or system for storing data, and includes databases and cloud storage.

[0696] "Learning" refers to the process by which a machine learning model extracts knowledge and patterns using new and historical data, thereby improving its accuracy and performance.

[0697] A "database" is a collection of data that has a structure that organizes information and allows for efficient access.

[0698] "Immediately assessing the degree of suspicion" refers to the process of quickly determining the reliability of the received communication number and content, and immediately estimating the possibility of fraud.

[0699] The present invention involves deploying a system that combines communication equipment, servers, and user terminals to effectively protect users from fraudulent phone calls. It primarily implements functions such as voice information analysis, fraudulent activity evaluation, automated response generation, speech synthesis output, and fraud pattern learning.

[0700] The server utilizes speech recognition and natural language processing technologies to process audio information received through communication devices. Specifically, the server uses speech recognition software (e.g., a common speech recognition API) to convert audio into text and then evaluates the likelihood of fraudulent activity based on that text. This evaluation employs machine learning models to compare the received phone number with past suspicious cases registered in the database.

[0701] The user terminal receives instructions from the server and automatically responds. The server uses a generative AI model to generate natural-sounding dialogue based on the prompt text. An example of a prompt text might be, "Generate a response that will reassure an elderly person in a fraudulent phone call." This generated response is output as actual voice on the user terminal using speech synthesis technology. An existing speech synthesis engine is used for speech synthesis.

[0702] Such a system allows, for example, elderly users who become targets of phone scams to deal with the situation safely without having to come into contact with the scammers. All conversational information is recorded on a server, and this data is used for the continuous learning of the generating AI model. This improves the overall fraud detection capability of the system and allows it to quickly adapt to new fraud patterns.

[0703] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0704] Step 1:

[0705] When a user terminal receives an incoming call, it obtains the phone number and caller information and sends it to the server. This input includes the received phone number and voice data. The user terminal encrypts this data and sends it to the server. This ensures the protection of personal information while enabling rapid analysis.

[0706] Step 2:

[0707] The server analyzes the received audio data. The input audio data is converted into text data using speech recognition technology. Next, the server compares the phone number with the database and evaluates whether it falls under the suspicious list. This comparison allows for an immediate determination of the degree of suspicion and an assessment of the possibility of fraudulent activity.

[0708] Step 3:

[0709] If the server determines that a call is likely to be a scam, it sends a prompt to the AI ​​model. The prompt might say, "Generate a reassuring response for an elderly person in a scam call," and the AI ​​model generates a natural language dialogue. This generation process is optimized based on past data and similar cases.

[0710] Step 4:

[0711] The generated response is sent from the server to the user's terminal, which outputs it as actual voice using speech synthesis technology. This output uses speech synthesis that has a sound quality close to that of a human voice. The terminal continues to interact with the scammer, adjusting the volume and tone to ensure the conversation proceeds naturally.

[0712] Step 5:

[0713] The server records the content of conversations with scammers and stores it in a database. The recorded conversation information is used in subsequent learning processes. The server updates the generated AI model based on the stored data and continues to learn in order to detect new scam patterns. This improves the overall scam detection capability of the system.

[0714] (Application Example 1)

[0715] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0716] Traditional fraud prevention systems often only detect fraudulent activity using fixed methods, limiting their ability to combat the ever-evolving nature of fraud. Furthermore, these systems require frequent updates, leading to user-intensive operation and difficulties in responding immediately to new fraud techniques. Additionally, insufficient awareness campaigns about fraudulent calls targeting the elderly, coupled with the inability to respond in real-time, raise concerns about the potential for widespread victimization.

[0717] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0718] In this invention, the server includes means for analyzing voice data received by a communication device and determining the possibility of fraudulent activity, means for activating an automated response device and generating a response using natural language processing technology, and means for outputting the generated response using speech synthesis technology. This enables real-time detection and immediate response to fraudulent activity, ensuring safety for the elderly and other users. Users can receive immediate fraud warnings on their devices through a pre-installed mobile application. Furthermore, by linking with cloud services, it is possible to respond quickly to various fraud patterns, reduce the complexity of user operation, and flexibly respond to the latest fraudulent activities.

[0719] A "communication device" is a device that receives voice data, analyzes that data, and has the function of determining the possibility of fraudulent activity.

[0720] An "automated response device" is a device that can speak responses generated using natural language processing technology to fraudsters.

[0721] "Natural language processing technology" is a technology for analyzing, understanding, and generating human language, enabling smooth communication between machines and humans.

[0722] "Speech synthesis technology" is a technology that converts text data into speech data and outputs it as a natural-sounding human voice.

[0723] A "database" is a storage device for saving conversation data with scammers, allowing for quick retrieval and reference of data as needed.

[0724] A "mobile application" is software that runs on a user's mobile device to display the results of fraud detection and provide warnings.

[0725] "Cloud services" utilize computing resources and data storage provided via the internet, and serve as a foundation for rapidly transmitting and updating the results of fraud detection.

[0726] A "user interface" is an interface that allows a user to interact with a computer system, and is designed for displaying and inputting information.

[0727] The implementation of this invention primarily involves a system in which a server, a terminal, and a user cooperate to prevent fraudulent phone calls. In its specific form, the invention begins with a step in which a communication device analyzes received voice data to determine the possibility of fraudulent activity. The server utilizes a communication device for this determination, comparing the received phone number and voice data with a database via a cloud service to evaluate the likelihood of fraud.

[0728] Based on the detection results, if fraudulent activity is detected, the server automatically activates an automated response system. This system uses natural language processing technology to generate an appropriate response and speaks it to the fraudster via speech synthesis technology. This allows for secure data collection while maintaining a call with the fraudster.

[0729] Simultaneously, a mobile application runs on the device, and when fraudulent activity is detected, the results are sent via a cloud service, and the user is warned through the user interface. This process allows users to quickly recognize fraudulent activity and prevent becoming a victim.

[0730] As a concrete example, let's assume a user has this system installed on their smartphone and receives a call. The system immediately identifies the number "080-XXXX-XXXX" as a scam and responds with an automated voice message saying, "I'm sorry, but could you please tell me your name again?" During this time, the user receives a real-time warning on their mobile device and can take the necessary action.

[0731] Examples of prompts for a generative AI model include the following:

[0732] A scam call has been identified. Please ask the scammer the following question: "Excuse me, but could you please tell me your name again?"

[0733] This embodiment allows for flexible responses to the latest fraudulent methods and provides users with peace of mind.

[0734] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0735] Step 1:

[0736] The server obtains voice data and phone numbers received from the user's terminal via a communication device. The voice data, as input data, is subjected to a voice analysis algorithm, and the phone numbers are compared against a database in the cloud. This process calculates the likelihood of fraudulent activity.

[0737] Step 2:

[0738] The server determines the likelihood of fraud based on the phone number and the results of voice analysis. If it determines that there is a high probability of fraud, it retrieves the relevant information from the database and generates a trigger to activate the automated response system. The input in this step is the phone number and the analysis results, and the output is the fraud determination result.

[0739] Step 3:

[0740] The device receives the fraud detection result from the server and notifies the user through the mobile application. The application displays a warning message using the user interface and prompts the user to take action. The input is the detection result from the server, and the output is the warning display.

[0741] Step 4:

[0742] If the server determines that a user is a scammer, it uses an automated response system to generate a prompt message to respond to the scammer. This prompt message is generated using natural language processing technology and output in real time using speech synthesis technology. The input is the judgment result and the generating AI model, and the output is the synthesized response.

[0743] Step 5:

[0744] The server records the call data with the scammer and stores it in a database so that the AI ​​model can learn new scam patterns at a later date. In this step, the call data is used as input, and the output is the stored training data.

[0745] These processing steps enable the system to detect fraud in real time and notify users, providing a systematic approach to combating fraudulent activity.

[0746] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0747] The system of the present invention is comprised of a communication device, a server, a user terminal, and an emotion engine. The user terminal first detects an incoming call and transmits the corresponding voice data to the server and the emotion engine. The server captures the incoming number and voice data and determines the possibility of fraudulent activity by comparing it with a database. Based on this determination, it activates an automated response system and generates an appropriate response using natural language processing technology.

[0748] Furthermore, the emotion engine analyzes the voice data to recognize the user's emotional state. It extracts the type and intensity of the emotion and sends it to the server. The server then takes this emotional information into account and adjusts the response accordingly. For example, if the user is showing anxiety or confusion, the response can be changed to a gentler tone to provide reassurance.

[0749] The generated response is converted into speech using speech synthesis technology and spoken to the other party. The server records all resulting interactions with the scammer and stores them in a database as more detailed data. This allows the AI ​​model to use the emotional responses as training data for learning scam patterns.

[0750] For example, when a user receives a fraudulent phone call, the server checks the number and recognizes the possibility of fraud. If the emotion engine determines from the user's voice that they are remaining calm, the server generates a normal response accordingly, resulting in a natural conversation. On the other hand, if the emotion engine determines that the user is showing signs of anxiety, it adjusts the response to provide reassurance, such as, "It's okay, let's talk slowly."

[0751] This system enhances protection against fraudulent activities and ensures the emotional safety of users. Through these processes, users can be effectively protected from fraudulent phone calls.

[0752] The following describes the processing flow.

[0753] Step 1:

[0754] The device detects an incoming call. It sends the incoming call information and voice data to the server and the emotion engine.

[0755] Step 2:

[0756] The server compares the incoming number against a database of past fraudulent numbers to determine the likelihood of fraud. If the result indicates a high probability of fraud, the automated response system activation flag is set.

[0757] Step 3:

[0758] The emotion engine analyzes the received audio data and extracts the user's emotional state. It generates data about the type and intensity of the emotion and sends it to the server.

[0759] Step 4:

[0760] The server receives emotional data from the emotion engine and uses natural language processing techniques to adjust the response. For example, if the user is expressing anxiety, the response is modified to be more reassuring.

[0761] Step 5:

[0762] The server sends the generated response to the terminal. The terminal then uses speech synthesis technology to convert this response into speech.

[0763] Step 6:

[0764] The device speaks a voiced response to the scammer. The voice tone and tempo are set to mimic an elderly person's voice, reducing any unnaturalness.

[0765] Step 7:

[0766] The server continuously records conversations with scammers and stores all data in a database. This includes conversation content and user sentiment data.

[0767] Step 8:

[0768] Once the server determines that the conversation has ended, it uses the recorded data as training material for the AI ​​model. This improves its ability to respond to new fraud patterns and emotional responses.

[0769] Step 9:

[0770] The server notifies the user when the conversation has ended and, if necessary, reports a summary to confirm that the scam call was handled properly.

[0771] (Example 2)

[0772] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0773] Conventional fraud prevention systems rely on simple methods such as number matching to identify fraud, making it difficult to detect when fraudsters change their numbers. Furthermore, they lack systems that take into account the emotional reactions of users, which hinders their ability to enhance user confidence.

[0774] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0775] In this invention, the server includes means for analyzing audio data received by a communication device and determining the possibility of fraudulent activity; means for activating an automated response device and generating a response using natural language processing technology based on the determination result; and means for outputting the generated response using speech synthesis technology. This makes it possible to highly detect fraudulent activity through audio data, and at the same time enhance the user's sense of security and strengthen the defense against fraudulent activity by providing a response that takes into account the user's emotional state.

[0776] A "communication device" is a device that has the function of sending and receiving voice data, and usually has an interface for exchanging data with other devices via a network.

[0777] "Audio data" refers to data used to record and process audio in digital format, and includes the content of a call and its voice patterns.

[0778] "Fraud detection" is a process of evaluating the likelihood of fraud based on received data and detecting such activity.

[0779] An "automatic response device" is a device that autonomously generates a response through a program and provides that response to the user or other party in real time.

[0780] "Natural language processing technology" is a technology that understands, analyzes, and generates human language, making it possible for machines to communicate with humans in a natural way.

[0781] "Speech synthesis technology" is a technology that converts text data into speech data and is used to generate natural-sounding speech.

[0782] "Emotional state" refers to a person's psychological state inferred from their voice or text, and includes emotions such as anxiety, relief, and anger.

[0783] "Response content" refers to the content of natural-sounding conversations and messages generated based on the received data and emotional state.

[0784] A "database" is a structured data storage system for efficiently storing, managing, and accessing information.

[0785] A "generative AI model" is an algorithm that uses machine learning techniques to learn patterns from large amounts of data and then makes predictions or generates new data.

[0786] The present invention is implemented as a system combining a communication device, a computer server, a user terminal, and an emotion engine. This system aims to detect fraudulent calls and optimize user responses, and is realized through the use of various technologies.

[0787] First, when the user terminal detects an incoming call, it collects voice data in real time and transmits it to a computer server and emotion engine via the internet. The user terminal uses a secure protocol to ensure the safe transfer of data.

[0788] Next, the server uses a specialized algorithm to analyze the voice data and the incoming number. To determine the likelihood of fraud, it performs number verification in conjunction with a database. If a number registered as a fraudulent number is detected, the server activates an automated response system. Using natural language processing technology, it generates a response based on the received information, and then uses speech synthesis technology to convert that response into speech.

[0789] The emotion engine uses a voice analysis algorithm to extract and recognize the user's emotional state in real time from voice data. For example, if anxiety is detected from the voice tone or speed, that emotional information is sent to the server. The server then adjusts the response based on that emotional information to provide the user with a reassuring response.

[0790] For example, when a user receives a scam call, the server is configured to respond in a gentle tone, such as "Don't worry, we'll support you," if the emotion engine reports that the user is feeling anxious.

[0791] An example of a prompt message would be, "If a user receives a fraudulent phone call, please tell me how to respond if they express anxiety." This is used to instruct the generating AI model on specific actions to take.

[0792] This system consists of communication devices, terminals, and specialized software running on servers, and functions to prevent fraudulent activities and enhance user confidence. In this way, users can handle phone calls more safely and with greater peace of mind.

[0793] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0794] Step 1:

[0795] Incoming call detection and data collection

[0796] The terminal detects incoming calls. Using a communication device API, it acquires the incoming number and voice data in real time. Inputs include the incoming call event, phone number, and voice signal, while output is the acquired voice data, prepared for transmission to the server and emotion engine.

[0797] Step 2:

[0798] Sending audio data

[0799] The terminal transmits the received voice data to the server and emotion engine via a secure communication protocol (e.g., HTTPS). Inputs include voice data and the incoming number, while output is the transmission of this data to the server and emotion engine.

[0800] Step 3:

[0801] Identifying fraudulent activity

[0802] The server uses the received voice data and incoming number to determine the possibility of fraud. It accesses a database and compares the incoming number against a list of known fraudulent numbers. The input is the incoming number and voice data, and the output is either a flag indicating the possibility of fraud or a status of "not found".

[0803] Step 4:

[0804] Analysis of emotional states

[0805] The emotion engine analyzes voice data to determine the user's emotional state. It analyzes voice tone, pitch, speed, etc., to extract emotional information. The input is voice data, and the output is the type of emotion (e.g., anxiety, relief) and its intensity.

[0806] Step 5:

[0807] Generating the response content

[0808] The server considers the presence or absence of fraudulent activity and sentiment information, and uses a generative AI model to create an appropriate response. The prompt in this case is in the form of, "If the user is showing anxiety, please generate an appropriate response." The input consists of a discrimination flag and sentiment data, and the output is the generated text of an optimized response.

[0809] Step 6:

[0810] Speech synthesis and responsive speech

[0811] The server uses speech synthesis technology to convert the generated response into speech and automatically plays it during the call with the scammer. The input is the generated text data, and the output is the generated audio data that is spoken to the other party on the call.

[0812] Step 7:

[0813] Data recording and storage

[0814] The server records and stores data in a database, including all interactions related to fraudulent activity and the emotional responses at the time. Inputs include conversation log data and emotional information, while output is this information stored in a structured format, making it available as subsequent training data.

[0815] (Application Example 2)

[0816] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0817] Conventional fraud prevention systems focus on identifying fraudulent activity through the analysis of voice data, but they have the drawback of lacking consideration for the user's emotional state. In particular, when a user experiences anxiety or confusion due to a fraudulent call, the system may not be able to respond appropriately, potentially compromising the user's emotional safety.

[0818] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0819] In this invention, the server includes means for analyzing audio information received by communication equipment and determining the possibility of fraudulent activity; means for analyzing the emotional state from the received audio information and adjusting the response content based on the user's emotions; and means for recording the conversation record with the fraudster in a storage device and utilizing it as a source of information for learning new fraudulent activity patterns. This makes it possible to strengthen the defense against fraudulent activity and ensure the emotional safety of the user.

[0820] "Communication equipment" is a general term for devices and terminals used to receive and analyze voice information.

[0821] "Audio information" refers to data captured by communication devices as audio signals, and is used as material for analysis and response generation.

[0822] A "discrimination method" is a system for evaluating and judging the possibility of fraudulent activity based on audio information.

[0823] An "automatic response device" is a system that generates an appropriate response based on the discrimination result.

[0824] "Natural language processing technology" is a technology that allows computers to understand and generate human language, and is used to generate response content.

[0825] "Emotional state" refers to the type and intensity of the user's emotions, as analyzed from their voice information.

[0826] "Response content" refers to the response to the user generated based on the analysis of the voice information.

[0827] "Speech synthesis technology" is a technology that converts generated responses into speech and outputs them.

[0828] A "conversation record" is data that records and saves the content of conversations with a scammer.

[0829] A "storage device" is a medium for storing conversation records and using them for later learning and analysis.

[0830] "Information sources" refer to data that provides the necessary information for learning about new fraudulent activities, and include conversation records, etc.

[0831] The system for carrying out this invention consists of a communication device, a server, an emotion recognition engine, an automatic response device, and a speech synthesis device. The communication device is responsible for receiving voice information from the user and transmitting it to the server. The server analyzes the received voice information and first determines the possibility of fraudulent activity. This determination is made by comparing the contact number and voice data with a database of past fraudulent activities. Furthermore, the server uses the emotion recognition engine to analyze the user's emotional state from the voice information. The emotion recognition engine evaluates the user's emotions based on parameters such as voice waveform, frequency, and speed.

[0832] The server then generates the optimal response that the automated response system should provide to the user, based on the likelihood of fraud and the user's emotional state. Utilizing natural language processing technology, the response is tailored to soothe the user's emotions. The generated response is then converted into natural-sounding speech by a speech synthesizer and returned to the user via communication equipment.

[0833] Furthermore, all conversation records are recorded and stored in a storage device. The stored data is used as information for the server to learn new fraud patterns. This allows the system to continuously improve the accuracy of fraud detection.

[0834] For example, if a user receives a suspicious call, the server immediately analyzes the audio information and, if it determines that there is a high probability of fraud, it retrieves data indicating that the user's emotions are unstable. In response, a reassuring response such as "It's okay, let's talk slowly?" is generated. An example of a prompt to the generating AI model would be, "Analyze the audio data of this call and assess the possibility of a fraudulent transaction. If the user is feeling anxious, generate an appropriate support message."

[0835] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0836] Step 1:

[0837] Communication equipment receives voice information from a user. The input is the user's spoken voice, and the output is digitized voice data. Voice sampling is performed to convert the voice into a digital signal.

[0838] Step 2:

[0839] The server receives the audio data and analyzes it to determine the possibility of fraudulent activity. The input is the audio data from step 1, and the output is the evaluation result of the fraud possibility. The server refers to the database, matches the audio data and contact numbers, and performs fraud pattern matching.

[0840] Step 3:

[0841] The server activates an emotion recognition engine and analyzes the user's emotional state from the audio data. The input is audio data, and the output is the user's emotion evaluation result. Audio parameters (waveform, audio frequency, speed, etc.) are calculated to identify the emotion.

[0842] Step 4:

[0843] The server applies natural language processing techniques to generate the optimal response based on the discrimination result and emotional state. The input is the fraud discrimination result and the emotional evaluation result, and the output is the generated text response. A generative AI model performs the language generation process.

[0844] Step 5:

[0845] A speech synthesizer converts the generated text response into speech. The input is the generated text response, and the output is the synthesized speech. A speech synthesis engine converts text into speech.

[0846] Step 6:

[0847] The server generates audio and returns it to the user via the communication device. The input is synthesized speech, and the output is the audio provided to the user. The communication device plays the audio.

[0848] Step 7:

[0849] The server saves conversation records and stores them in a database. Inputs are log information and audio recordings from all processes, and output is the conversation data stored in the database. A storage solution is used for data storage.

[0850] Step 8:

[0851] The server learns new fraud patterns using stored data. The input is conversation records in the database, and the output is an updated fraud detection model. A machine learning algorithm performs the learning process.

[0852] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0853] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0854] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0855] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0856] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0857] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0858] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0859] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0860] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0861] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0862] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0863] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0864] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0865] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0866] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0867] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0868] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0869] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0870] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0871] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0872] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0873] The following is further disclosed regarding the embodiments described above.

[0874] (Claim 1)

[0875] A means of analyzing audio data received by a communication device to determine the possibility of fraudulent activity,

[0876] Based on the determination result, the system activates an automated response device and generates a response using natural language processing technology.

[0877] A means for outputting the generated response content using speech synthesis technology,

[0878] A means of recording conversation data with scammers and saving it in a database,

[0879] A means of learning new patterns of fraudulent activity based on saved conversation data,

[0880] A system that includes this.

[0881] (Claim 2)

[0882] The system according to claim 1, wherein the means for determining fraudulent activity compares the received telephone number with a database of past fraudulent activities.

[0883] (Claim 3)

[0884] The system according to claim 1, wherein the automated response device pretends to be an elderly person and generates conversations for the purpose of reducing the unnaturalness of the response.

[0885] "Example 1"

[0886] (Claim 1)

[0887] A means of analyzing audio information received by communication devices and evaluating the probability of fraudulent activity,

[0888] Based on the evaluation results, the system includes means for activating a response generation device and generating conversation content using natural language processing technology,

[0889] A means for outputting the generated conversation content using speech synthesis technology,

[0890] A means of recording and storing information about conversations with fraudsters in a storage device,

[0891] A means of learning new patterns of fraudulent activity based on saved dialogue information,

[0892] A means of immediately evaluating the degree of suspicion by comparing the incoming call number with a database,

[0893] A means of adjusting the voice output to continue the conversation using natural voice characteristics,

[0894] A system that includes this.

[0895] (Claim 2)

[0896] The system according to claim 1, wherein the means for evaluating the fraudulent activity is to compare it with a database of information on past fraudulent activities.

[0897] (Claim 3)

[0898] The system according to claim 1, wherein the response generating device generates conversations that mimic a specific age group and aim to reduce the unnaturalness of the voice.

[0899] "Application Example 1"

[0900] (Claim 1)

[0901] A means of analyzing audio data received by a communication device to determine the possibility of fraudulent activity,

[0902] Based on the determination result, the system activates an automated response device and generates a response using natural language processing technology.

[0903] A means for outputting the generated response content using speech synthesis technology,

[0904] A means of recording conversation data with scammers and saving it in a database,

[0905] A means of learning new patterns of fraudulent activity based on saved conversation data,

[0906] A means for running a mobile application on a user's mobile device to provide a warning about fraudulent activity,

[0907] A means of transmitting the results of fraud detection via a cloud service and notifying the user via a user interface,

[0908] A system that includes this.

[0909] (Claim 2)

[0910] The system according to claim 1, wherein the means for determining fraudulent activity compares the received telephone number with a database of past fraudulent activities.

[0911] (Claim 3)

[0912] The system according to claim 1, wherein the automated response device pretends to be an elderly person and generates conversations for the purpose of reducing the unnaturalness of the response.

[0913] "Example 2 of combining an emotion engine"

[0914] (Claim 1)

[0915] A means of analyzing audio data received by a communication device to determine the possibility of fraudulent activity,

[0916] Based on the determination result, the system activates an automated response device and generates a response using natural language processing technology.

[0917] A means for outputting the generated response content using speech synthesis technology,

[0918] In addition to identifying fraudulent activities, the system analyzes human emotional states from voice data and incorporates emotional information into responses.

[0919] A means of recording conversation data with scammers and saving it in a database,

[0920] A method for updating a generative AI model, including emotional responses, by learning new patterns of fraudulent behavior based on saved conversation data,

[0921] A system that includes this.

[0922] (Claim 2)

[0923] The system according to claim 1, wherein the means for determining fraudulent activity compares the received telephone number with a database of past fraudulent activities.

[0924] (Claim 3)

[0925] The system according to claim 1, wherein the automated response device adjusts the content of the response according to the emotional state of the person and generates a conversation in a way that gives a sense of security and trust.

[0926] "Application example 2 when combining with an emotional engine"

[0927] (Claim 1)

[0928] A means of analyzing audio information received by communication devices to determine the possibility of fraudulent activity,

[0929] Based on the determination result, the system activates an automated response device and generates a response using natural language processing technology.

[0930] A means for analyzing the emotional state from received audio information and adjusting the response content based on the user's emotions,

[0931] A means for outputting the generated response content using speech synthesis technology,

[0932] A means of recording conversations with fraudsters in a storage device and using it as a source of information to learn new patterns of fraudulent activity,

[0933] A system that includes this.

[0934] (Claim 2)

[0935] The system according to claim 1, wherein the means for determining fraudulent activity compares the received contact number with a source of information for past fraudulent activity.

[0936] (Claim 3)

[0937] The system according to claim 1, wherein the automated response device generates a conversation that takes into account the user's emotional state and aims to generate a natural response. [Explanation of symbols]

[0938] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means of analyzing audio data received by a communication device to determine the possibility of fraudulent activity, Based on the determination result, the system activates an automated response device and generates a response using natural language processing technology. A means for outputting the generated response content using speech synthesis technology, A means of recording conversation data with scammers and saving it in a database, A means of learning new patterns of fraudulent activity based on saved conversation data, A system that includes this.

2. The system according to claim 1, wherein the means for determining fraudulent activity is to compare the received telephone number with a database of past fraudulent activities.

3. The system according to claim 1, wherein the automated response device pretends to be an elderly person and generates conversations for the purpose of reducing the unnaturalness of the response.

Images