system

Abstract

We provide the system. [Solution] A means for acquiring the voice of a user during a call, A conversion means for converting audio data acquired by the aforementioned audio acquisition means into text data, An analysis means for analyzing the text data obtained by the conversion means and evaluating the possibility of fraud, A recording device for recording calls in the event that a potential fraud is detected, A notification system to inform users and registered family members of potential fraud, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of the chatbot's character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Damage caused by special fraud has become a major problem throughout society, and especially the elderly are often targeted. The methods of telephone fraud have become sophisticated, and effective measures to prevent damage are required. However, at present, means for analyzing the content of a call in real time to prevent damage are limited. In contrast, a system that can detect the possibility of fraud during a call and respond promptly is needed.

Means for Solving the Problems

[0005] This invention provides a system comprising: an audio acquisition means for acquiring user call audio; a conversion means for converting audio data into text data; an analysis means for analyzing text data to evaluate the possibility of fraud; a recording means for recording calls when the possibility of fraud is detected; and a notification means for informing the user and registered family members of the possibility of fraud. By analyzing keywords and phrases related to fraud using natural language processing technology, notifications are made in real time if there is a possibility of fraudulent activity, and the recorded call content is also shared with family members. As a result, it is possible to provide a means to prevent fraud from occurring for users, including the elderly.

[0006] "Voice acquisition means" refers to a system or device for capturing a user's call audio in real time and converting it into a processable data format.

[0007] "Conversion means" refers to a system or process that uses automatic speech recognition technology to convert audio data into text information.

[0008] "Analysis means" refers to a system or method that performs natural language processing to analyze text data and assess the likelihood of fraud.

[0009] "Recording means" refers to a system or device that records telephone conversations in digital format and makes them available for storage or transmission as needed.

[0010] "Notification means" refers to a system or method for sending information to users or registered family members to inform them of potential fraud.

[0011] A "natural language processing model" is a collection of algorithms and methods for computers to understand and process natural language used by humans. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine. [Modes for carrying out the invention]

[0013] Hereinafter, an example of an embodiment of the system relating to the technology of this disclosure will be described with reference to the attached drawings.

[0014] First, the terms used in the following description will be explained.

[0015] In the following embodiments, the labeled processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, the labeled RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the labeled storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0018] In the following embodiments, the labeled communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc.

[0019] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] For the implementation of this invention, a user terminal, a server, and a communication network connecting them are essential. In this system configuration, the terminal is equipped with voice acquisition means to acquire voice during a call in real time. The acquired voice data is converted into text data using automatic speech recognition (ASR) software on the terminal.

[0034] The converted text data is sent to the server via a communication network. The server uses natural language processing (NLP) techniques to analyze this text data and assess the likelihood of fraud. The analysis employs a model that meticulously searches for keywords and phrases related to fraud.

[0035] If a call is deemed potentially fraudulent, the server sends a notification to the device, and the device automatically records the call. The recorded audio data is then sent to the server, which sends a notification to the user and any pre-registered family members. This notification not only informs them of the potential fraud but also provides them with access to the recorded audio.

[0036] As a concrete example, consider a scenario where a user receives a phishing call. If the call contains suspicious keywords such as "bank account" or "password," the server immediately determines that it is highly likely to be a scam. The receiving device records the call, and a warning is sent to the user and their family via the server. This process allows the user and their family to identify the risk of fraud and take prompt action.

[0037] The following describes the processing flow.

[0038] Step 1:

[0039] The terminal detects when a user initiates a call and activates the voice acquisition mechanism to capture the call audio in real time.

[0040] Step 2:

[0041] The device transmits the acquired audio data to the Automatic Speech Recognition (ASR) system in real time, converting the audio into text data.

[0042] Step 3:

[0043] The terminal utilizes a communication module to send the converted text data to the server, transferring the text data to the server.

[0044] Step 4:

[0045] The server uses natural language processing (NLP) techniques to analyze the received text data and runs a model designed to assess whether it is potentially fraudulent.

[0046] Step 5:

[0047] If the server determines, based on its analysis, that there is a high probability of fraud, it generates a fraud alert and sends that information to the device.

[0048] Step 6:

[0049] The device receives a notification from the server, automatically records the relevant call, and generates a recording file.

[0050] Step 7:

[0051] The device transfers the recorded audio data to the server, and the server stores the recording file.

[0052] Step 8:

[0053] The server sends a notification to the user and their pre-registered family members, containing information about the potential for fraud and how to access the recording files.

[0054] (Example 1)

[0055] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0056] In modern communication, users are at increasing risk of becoming involved in fraudulent calls related to malicious activity. Because there is a lack of means to identify such calls in real time and effectively warn users, preventing fraud is difficult.

[0057] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0058] In this invention, the server includes means for acquiring acoustic data to capture user conversations, means for converting the acoustic data acquired by the acoustic data acquisition means into text data, and means for analyzing the converted text data to evaluate the possibility of fraudulent activity. This makes it possible to quickly identify calls related to fraudulent activity and to issue appropriate warnings to the user and their relatives.

[0059] "Acoustic data acquisition means" refers to a device or technology for acquiring user conversations in real time, and mainly includes microphones and related signal processing technologies.

[0060] "Means of converting into text data" refers to technologies that convert acquired audio data so that it can be treated as text information, and automatic speech recognition (ASR) technology falls under this category.

[0061] "Means of analysis" refers to techniques for analyzing converted character data and evaluating the likelihood of fraudulent activity, including using natural language processing (NLP) techniques to identify words and expressions related to fraud.

[0062] "Means of recording" means a device or process for saving a conversation in which potential misconduct has been detected, and includes techniques for electronically recording audio data.

[0063] "Means of notification" refers to methods of issuing warnings to users and registered relatives in the event of potential fraudulent activity, such as email or in-app notifications.

[0064] To implement this invention, a user's terminal, a server, and a communication network connecting them are required. The terminal is equipped with acoustic data acquisition means to acquire the user's conversation in real time. A microphone is used for this acquisition, and acoustic sensing technology built into smartphones and tablets is applied.

[0065] The device converts the acquired audio data into text data using automatic speech recognition (ASR) software. Commonly used software for this purpose includes speech-to-text technologies, such as Google® Speech-to-Text and Watson® Speech to Text by Google, Inc.

[0066] The terminal sends the converted character data to the server via the communication network. The server analyzes this character data using natural language processing (NLP) techniques to evaluate the likelihood of fraudulent activity. Python's natural language processing libraries, such as NLTK and spaCy, are used for the analysis. The server tokenizes the information and identifies relevant keywords to numerically analyze the likelihood of fraud.

[0067] When the server detects potential fraudulent activity, it notifies the terminal accordingly. Upon receiving the notification, the terminal immediately begins recording the conversation and generates audio data. This audio data is sent to the server as a compressed audio file. The server then issues a warning to the user and registered relatives, informing them of the potential fraud and providing information on how to access the audio data.

[0068] For example, in the case of a call that may be a phishing scam, the server will determine that the call is dangerous if certain keywords such as "bank account" or "password" are detected. Based on this, the user's device will automatically record the call and immediately notify the relevant parties. Prompts to be input into the generating AI model include "Please list keywords that indicate a phishing scam" and "Please explain the fraud detection process using natural language processing."

[0069] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0070] Step 1:

[0071] The terminal acquires user conversations in real time using an acoustic data acquisition method. The input is audio captured through the terminal's microphone. This input is acquired as a digital signal and prepared for subsequent processing.

[0072] Step 2:

[0073] The terminal converts acquired audio data into text data using automatic speech recognition (ASR) software. The input is the digital audio data obtained in step 1, which is then converted from an audio signal to text information in step 2. Specifically, the ASR engine analyzes the audio pattern and replaces it with the corresponding string. The output is the converted text data.

[0074] Step 3:

[0075] The terminal sends the converted character data to the server via the communication network. The character data from step 2 is used as input and is securely sent to the server using a network protocol (e.g., HTTPS). The output is the character data received by the server.

[0076] Step 4:

[0077] The server analyzes text data using natural language processing (NLP) techniques to assess the likelihood of fraudulent activity. The input is the text data received in step 3, and through data analysis, keywords related to fraud are identified. Specifically, the analysis algorithm tokenizes the words and scores the likelihood of fraud. The output is the evaluation result indicating the likelihood of fraudulent activity.

[0078] Step 5:

[0079] If the server determines that there is a high probability of fraudulent activity, it sends a notification to the device. The input is the evaluation result obtained in step 4, which is sent to the device as a warning message. The output is the notification displayed on the device.

[0080] Step 6:

[0081] The device automatically starts recording the conversation when it receives a notification. The input is the notification from step 5, which activates the device's recording function and saves the audio to a file. The output is the recorded audio file.

[0082] Step 7:

[0083] The terminal sends the recorded audio file to the server. The input is the audio file generated in step 6, which is then transferred to the server via the network in step 7. The output is the recorded data stored on the server.

[0084] Step 8:

[0085] The server re-analyzes the recorded audio data and notifies the user and pre-registered relatives of the results. The input is the recorded data received in step 7, and the notification is executed based on the final assessment of the likelihood of fraud. The output is a warning message sent to the user and relatives.

[0086] (Application Example 1)

[0087] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0088] In modern society, telephone-based fraud is on the rise. These methods are sophisticated, increasing the risk of many people becoming victims. Therefore, there is a need for a reliable system that can assess the possibility of fraud in real time and issue warnings quickly.

[0089] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0090] In this invention, the server includes: voice acquisition means for acquiring the user's voice; conversion means for converting the voice information acquired by the voice acquisition means into text information; analysis means for analyzing the text information obtained by the conversion means to evaluate the possibility of fraud; recording means for recording communications when the possibility of fraud is detected; notification means for notifying the user and registered family members of the possibility of fraud; and warning means for detecting the possibility of fraud in real time and sending a warning. This makes it possible to warn of fraudulent activity in advance and protect the user's information and property.

[0091] "Voice acquisition means" refers to a device or software for acquiring a user's voice as a digital signal.

[0092] "Conversion means" refers to a device or software that performs the process of converting acquired audio information into text information.

[0093] "Analysis means" refers to a device or software that utilizes natural language processing technology to evaluate the possibility of fraud based on textual information.

[0094] "Recording means" refers to a device or software that records relevant audio or text information when a potential fraud is detected.

[0095] "Notification means" refers to a means or device for communicating with users and their registered family members to inform them of the possibility of fraud.

[0096] A "warning device" is a device or software that detects potential fraud in real time and immediately issues a warning to the user.

[0097] The system for realizing this invention consists of a series of processes for acquiring, converting, analyzing, and notifying voice data. The system is implemented using a user's terminal, a server, and a communication network connecting them.

[0098] The device functions as a means of voice acquisition, capturing the user's call audio via its microphone. This audio data is converted into text by the device's built-in automatic speech recognition (ASR) software. Specifically, Google's speech recognition API is used.

[0099] The converted text information is transmitted to the server via a communication network. The server analyzes the text information using natural language processing (NLP) techniques. Specific NLP models are used to detect words and phrases related to fraudulent activity. This analysis assesses the likelihood of fraud.

[0100] If a potential scam is detected, the server will immediately issue a notification. Notification methods include push notifications and email. The user and their family members will be notified that the relevant communication has been recorded and that it may be fraudulent, and details of access to the recording will be provided if necessary.

[0101] For example, if a user receives a call containing phrases such as "credit card" or "password," the system immediately flags it as potentially fraudulent and notifies registered family members that "a potentially fraudulent call has been detected."

[0102] When using generative AI models, one possible prompt would be: "If FraudGuarder analyzes a suspicious call and determines it may be a scam, how will it notify the user and share the recorded call with their family?"

[0103] This allows users to receive warnings before they become victims of fraud and to take prompt action in the situation.

[0104] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0105] Step 1:

[0106] The device uses a microphone to acquire the user's call audio. The acquired analog audio signal is converted into a digital signal and input as audio data.

[0107] Step 2:

[0108] The device converts the acquired audio data into text data using automatic speech recognition (ASR) software, specifically Google's speech recognition API. The input is audio data, and the output is text information.

[0109] Step 3:

[0110] The converted character information is sent to the server via the communication network. The input is character information, and the data is sent to the server.

[0111] Step 4:

[0112] The server uses natural language processing (NLP) techniques to analyze textual information. Specifically, it uses a model to detect words and phrases related to fraud and analyzes the input textual information. The output is an evaluation result indicating the likelihood of fraud.

[0113] Step 5:

[0114] If the server determines, based on the analysis, that there is a high probability of fraud, it generates a warning and notifies the terminal. The input is the analysis result, and the output is sent to the user as a warning message.

[0115] Step 6:

[0116] The device records and saves the corresponding call upon receiving a notification from the server. The input is the notification message, and the output is the saving of the recorded data.

[0117] Step 7:

[0118] The server uses notification mechanisms to alert users and registered family members to potential fraud, granting them access to the recorded data. Inputs are the recorded data and access rights information, while output is sent to users and family members as notification messages.

[0119] By following these steps, users can detect the risk of fraud in advance and take prompt action.

[0120] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0121] This invention provides a system that combines a user's call audio acquisition and fraud analysis with an emotion engine that recognizes the user's emotions. A voice acquisition means is installed on the terminal to capture the user's call in real time. The voice data is converted into text data by a conversion means and then transmitted to a server via a communication network.

[0122] The server analyzes the received text data using an analysis tool. This analysis employs a natural language processing (NLP) model to identify keywords and phrases related to fraud. Furthermore, the analysis tool works in conjunction with an emotion engine to analyze changes in the user's voice tone and pitch to determine their emotional state. The emotion engine incorporates emotional information as a corrective factor into the fraud evaluation process, thereby determining the likelihood of fraud with greater accuracy.

[0123] If a call is deemed highly likely to be fraudulent, the server will notify the device to record the call. Furthermore, if the emotion engine determines that the user's emotional state is outside the normal range, a notification will be sent immediately. This notification, including information on accessing the recording file, will be provided to the user and any pre-registered family members.

[0124] As a concrete example, consider a case where a user's voice is trembling unnaturally during a phone call. In this case, the emotion engine can detect this abnormal emotional change and generate a warning overlaid on the possibility of fraud. This warning is immediately sent to the user and their family, allowing for further action after reviewing the recorded call. This system provides users with a highly reliable means of dealing with the risk of fraud.

[0125] The following describes the processing flow.

[0126] Step 1:

[0127] The device detects when a user initiates a call, activates the voice acquisition mechanism, and captures the call audio in real time.

[0128] Step 2:

[0129] The audio data acquired by the device is converted into text data using an automatic speech recognition (ASR) system.

[0130] Step 3:

[0131] The terminal sends the converted text data to the server.

[0132] Step 4:

[0133] The server analyzes the received text data and performs keyword analysis using a natural language processing (NLP) model.

[0134] Step 5:

[0135] The server utilizes an emotion engine to analyze emotional information extracted from the voice (e.g., changes in voice tone and pitch) and determine the user's emotional state.

[0136] Step 6:

[0137] Based on the analysis results, the server comprehensively evaluates the likelihood of fraud and the emotional state, and calculates an overall score.

[0138] Step 7:

[0139] The server notifies the device to record calls that it determines are highly likely to be fraudulent or involve an abnormal emotional state.

[0140] Step 8:

[0141] The device records the specified call and sends the recording data to the server.

[0142] Step 9:

[0143] The server stores the recorded audio data and sends notifications of fraud risk or emotional disturbances to the user and pre-registered family members, and provides information on how to access the recording files.

[0144] (Example 2)

[0145] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0146] Fraudulent activities are becoming more sophisticated over time, making detection difficult with conventional methods. Furthermore, there is a need for measures to prevent attempted fraud in advance by detecting anomalies based on changes in user emotions. Conventional systems cannot interpret emotions from voice, making it difficult to accurately assess the likelihood of fraud.

[0147] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0148] In this invention, the server includes an audio acquisition means for acquiring the user's call audio, a conversion means for converting the audio data into text data, an analysis means for checking keywords and phrases related to fraud, an emotion analysis means for analyzing emotional information contained in the user's voice, an integrated evaluation means for integrating the analysis results to determine the likelihood of fraud with high accuracy, a recording means for recording the relevant call, and a notification means for issuing a notification. This enables early detection of fraud risk and prompt warning to the user.

[0149] "Voice acquisition means" refers to a device or function for collecting a user's call audio in real time.

[0150] "Conversion means" refers to the process or technology used to convert acquired audio data into text data.

[0151] "Analysis means" refers to a function that analyzes text data and identifies keywords and phrases related to fraud.

[0152] "Emotional analysis methods" refer to technologies for extracting and analyzing emotional information from user voice data.

[0153] The "integrated evaluation method" is a function that integrates analysis results and emotional information to evaluate the likelihood of fraud with high accuracy.

[0154] "Recording means" refers to technology or equipment for recording potentially fraudulent phone calls.

[0155] "Notification methods" refer to features that inform users and their registered family members of potential fraud.

[0156] The system of the present invention has the function of acquiring a user's call audio in real time and highly evaluating the possibility of fraud. When a user starts a call, the terminal captures audio data using an audio acquisition means. This audio data is converted into text data using a conversion means within the terminal. Speech recognition software (e.g., speech recognition API) is used for this conversion.

[0157] The converted text data is sent to the server via a communication network. The server analyzes the text data using a natural language processing model to identify keywords and phrases related to fraud. The server also extracts emotional information from the user's voice using sentiment analysis and uses this information to evaluate the fraud.

[0158] If a call is deemed to be at high risk of fraud, or if the user's emotional state is judged to be outside the normal range, the server will instruct the recording of the call and send a notification to the device containing information on how to access the recording data. Furthermore, this information will be notified to the user and pre-registered family members to enable a swift response.

[0159] For example, a user might receive a call from an unfamiliar caller who emphasizes urgency or speaks in an unstable tone. In this case, the server detects the tone of voice based on sentiment analysis and determines that it is likely to be a scam. The user and their family can then review the recording of the call and take appropriate action if necessary.

[0160] Example prompt for a generative AI model: "Detect changes in the user's emotions during a call and assess the risk of fraud."

[0161] This invention allows users to communicate with greater peace of mind and reduces the risk of becoming a victim of fraud.

[0162] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0163] Step 1:

[0164] When a user initiates a call, the device captures audio data in real time using an audio acquisition device. At this time, the user's call audio is the input, and audio data is generated as the output. Specifically, the device uses a microphone to acquire the audio waveform as digital data.

[0165] Step 2:

[0166] The terminal converts the acquired audio data into text data using a conversion mechanism. Here, the input is audio data, and the output is the corresponding text data. This process utilizes speech recognition software, which analyzes the pattern of the audio waveform and represents its content as text.

[0167] Step 3:

[0168] The terminal sends the converted text data to the server via the communication network. The input is text data, and the output exists as data transferred to the server. In this process, data is encrypted for security purposes to prevent unauthorized access.

[0169] Step 4:

[0170] The server processes the received text data using parsing tools to identify keywords and phrases related to fraud. The input for the parsing is the text data that reaches the server, and the output is an evaluation result indicating the likelihood of fraud. Specifically, a natural language processing model is used to detect specific words and syntax in the data.

[0171] Step 5:

[0172] The server uses emotion analysis tools in parallel with the analysis to extract emotional information from the user's voice. The input to this analysis is variables obtained from the voice data (e.g., tone, pitch), and the output is an evaluation result indicating the user's emotions. Emotion recognition software is used to analyze the voice parameters and identify changes in emotion.

[0173] Step 6:

[0174] The server integrates the analysis results and sentiment analysis results using an integrated evaluation system to determine the likelihood of fraud with high accuracy. The input to this integration is two evaluation results: analysis and sentiment, and the output is an overall fraud risk assessment value. This enables more reliable fraud detection.

[0175] Step 7:

[0176] If a call is deemed highly likely to be fraudulent, the server instructs the device to record the call and sends a notification containing information on how to access the recording data. The input is a fraud risk assessment value, and the output is notification information for the user and pre-registered family members. The server generates an access link to the recording data and includes it in the notification message to ensure rapid information dissemination.

[0177] (Application Example 2)

[0178] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0179] In recent years, with advancements in communication technology, fraudulent activities exploiting voice calls have increased. Therefore, there is a need for a system that allows users to quickly detect potential fraud during a call and take countermeasures. However, conventional fraud detection systems rely solely on keyword and phrase detection, which can result in insufficient accuracy. Furthermore, simply notifying users of potential fraud can worsen their emotional state. Therefore, a system is needed that analyzes changes in the user's emotional state and utilizes this information for fraud detection, enabling more accurate notifications.

[0180] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0181] In this invention, the server includes means for acquiring sound, means for converting it into text information, and means for analyzing emotions to determine the emotional state. This enables a more accurate assessment of the possibility of fraud, taking into account not only the possibility of fraud during a call but also the user's emotional fluctuations. This system also provides support for users to make calls with peace of mind by quickly notifying them of any abnormalities in their emotional state.

[0182] The "acoustic acquisition means" is a device that captures voice information from a user's call in real time and provides it as data necessary for subsequent processing.

[0183] A "conversion means" is a device that has the function of converting acquired audio information into text information, making it available as basic data for analysis.

[0184] An "analysis tool" is a device that analyzes textual information to assess the likelihood of fraud and makes a judgment based on specific conditions.

[0185] An "emotional analysis tool" is a device that has the function of understanding the emotional state of a user during a call and analyzing its changes to evaluate the user's mental state.

[0186] A "recording device" is a device that has the function of recording phone calls in response to the detection of potential fraud or abnormal emotional states, and saving them in a format that can be reviewed at a later date.

[0187] A "notification method" is a function that quickly informs users and registered parties of potential fraud or abnormal emotional states that have been detected.

[0188] The system implementing this invention primarily processes voice data during a call in real time and has the function of evaluating the possibility of fraud and the user's emotional state. The server converts the voice data sent from the user's terminal into text information using the Google Speech-to-Text API. This makes it possible to transform the voice information into a format that is easy to analyze.

[0189] The converted text information is sent to the server, where it is analyzed using a natural language processing model (e.g., BERT or GPT). The purpose of the analysis is to identify vocabulary and expressions related to fraud and to assess the likelihood of fraud. In addition, a sentiment analysis engine such as IBM Watson Tone Analyzer can be used to simultaneously determine the user's emotional state, and if an abnormal change in emotion is detected, the fraud likelihood assessment can be corrected.

[0190] If the server detects potential fraud or emotional instability, it will send alerts to the user and registered stakeholders using notification methods such as Firebase Cloud Messaging. This allows users to quickly recognize the situation and take appropriate action.

[0191] For example, if a user is discussing a real estate transaction over the phone and the other party speaks in an unnatural tone that seems to be pressuring them to make an immediate decision, the system will detect this and analyze the emotional changes that indicate the user's anxiety is increasing. Based on this information, a potential fraud alert will be sent immediately.

[0192] An example of a prompt message is: "Assess the likelihood of fraud based on the user's call content. Also, analyze the user's emotional changes during the call and provide details if anything seems unnatural."

[0193] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0194] Step 1:

[0195] The terminal captures the user's call audio using an acoustic acquisition device. The input is raw audio data, and the output is real-time captured audio data. Specifically, the terminal's microphone collects the audio and converts it into a digital signal.

[0196] Step 2:

[0197] The device converts captured audio data into text information using a conversion mechanism. The input is audio data, and the output is text data. Specifically, the Google Speech-to-Text API is used to analyze the audio data and generate the corresponding text.

[0198] Step 3:

[0199] The server uses analysis tools to evaluate the likelihood of fraud in text data sent from a terminal. The input is converted character information, and the output is the evaluation result regarding the likelihood of fraud. Specifically, a natural language processing model (such as BERT or GPT) analyzes the text and detects vocabulary and expressions related to fraud.

[0200] Step 4:

[0201] The server analyzes the text information and uses sentiment analysis tools to determine the user's emotional state. The input is text data, and the output is the evaluation result of the emotional state. Specifically, IBM Watson Tone Analyzer detects emotional fluctuations from the text and analyzes the results.

[0202] Step 5:

[0203] The server integrates the fraud likelihood assessment and the emotional state assessment to make a final decision. The input is the output results from steps 3 and 4, and the output is the final fraud likelihood assessment and the necessary actions. Specifically, a fraud warning alert is generated based on these two evaluation results.

[0204] Step 6:

[0205] The server uses notification methods to send alerts to users and registered stakeholders. The input is the final fraud warning alert, and the output is the notification sent to users and stakeholders. Specifically, Firebase Cloud Messaging is used to share information by displaying the alert on stakeholders' devices immediately.

[0206] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0207] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0208] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0209] [Second Embodiment]

[0210] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0211] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0212] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0213] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0214] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0215] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0216] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0217] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0218] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0219] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0220] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0221] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0222] For the implementation of this invention, a user terminal, a server, and a communication network connecting them are essential. In this system configuration, the terminal is equipped with voice acquisition means to acquire voice during a call in real time. The acquired voice data is converted into text data using automatic speech recognition (ASR) software on the terminal.

[0223] The converted text data is sent to the server via a communication network. The server uses natural language processing (NLP) techniques to analyze this text data and assess the likelihood of fraud. The analysis employs a model that meticulously searches for keywords and phrases related to fraud.

[0224] If a call is deemed potentially fraudulent, the server sends a notification to the device, and the device automatically records the call. The recorded audio data is then sent to the server, which sends a notification to the user and any pre-registered family members. This notification not only informs them of the potential fraud but also provides them with access to the recorded audio.

[0225] As a concrete example, consider a scenario where a user receives a phishing call. If the call contains suspicious keywords such as "bank account" or "password," the server immediately determines that it is highly likely to be a scam. The receiving device records the call, and a warning is sent to the user and their family via the server. This process allows the user and their family to identify the risk of fraud and take prompt action.

[0226] The following describes the processing flow.

[0227] Step 1:

[0228] The terminal detects when a user initiates a call and activates the voice acquisition mechanism to capture the call audio in real time.

[0229] Step 2:

[0230] The device transmits the acquired audio data to the Automatic Speech Recognition (ASR) system in real time, converting the audio into text data.

[0231] Step 3:

[0232] The terminal utilizes a communication module to send the converted text data to the server, transferring the text data to the server.

[0233] Step 4:

[0234] The server uses natural language processing (NLP) techniques to analyze the received text data and runs a model designed to assess whether it is potentially fraudulent.

[0235] Step 5:

[0236] If the server determines, based on its analysis, that there is a high probability of fraud, it generates a fraud alert and sends that information to the device.

[0237] Step 6:

[0238] The device receives a notification from the server, automatically records the relevant call, and generates a recording file.

[0239] Step 7:

[0240] The device transfers the recorded audio data to the server, and the server stores the recording file.

[0241] Step 8:

[0242] The server sends a notification to the user and their pre-registered family members, containing information about the potential for fraud and how to access the recording files.

[0243] (Example 1)

[0244] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0245] In modern communication, users are at increasing risk of becoming involved in fraudulent calls related to malicious activity. Because there is a lack of means to identify such calls in real time and effectively warn users, preventing fraud is difficult.

[0246] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0247] In this invention, the server includes means for acquiring acoustic data to capture user conversations, means for converting the acoustic data acquired by the acoustic data acquisition means into text data, and means for analyzing the converted text data to evaluate the possibility of fraudulent activity. This makes it possible to quickly identify calls related to fraudulent activity and to issue appropriate warnings to the user and their relatives.

[0248] "Acoustic data acquisition means" refers to a device or technology for acquiring user conversations in real time, and mainly includes microphones and related signal processing technologies.

[0249] "Means of converting into text data" refers to technologies that convert acquired audio data so that it can be treated as text information, and automatic speech recognition (ASR) technology falls under this category.

[0250] "Means of analysis" refers to techniques for analyzing converted character data and evaluating the likelihood of fraudulent activity, including using natural language processing (NLP) techniques to identify words and expressions related to fraud.

[0251] "Means of recording" means a device or process for saving a conversation in which potential misconduct has been detected, and includes techniques for electronically recording audio data.

[0252] "Means of notification" refers to methods of issuing warnings to users and registered relatives in the event of potential fraudulent activity, such as email or in-app notifications.

[0253] To implement this invention, a user's terminal, a server, and a communication network connecting them are required. The terminal is equipped with acoustic data acquisition means to acquire the user's conversation in real time. A microphone is used for this acquisition, and acoustic sensing technology built into smartphones and tablets is applied.

[0254] The device converts the acquired audio data into text data using automatic speech recognition (ASR) software. Commonly used software for this purpose includes speech-to-text technologies, such as Google Speech-to-Text and Watson Speech-to-Text from Google Inc.

[0255] The terminal sends the converted character data to the server via the communication network. The server analyzes this character data using natural language processing (NLP) techniques to evaluate the likelihood of fraudulent activity. Python's natural language processing libraries, such as NLTK and spaCy, are used for the analysis. The server tokenizes the information and identifies relevant keywords to numerically analyze the likelihood of fraud.

[0256] When the server detects potential fraudulent activity, it notifies the terminal accordingly. Upon receiving the notification, the terminal immediately begins recording the conversation and generates audio data. This audio data is sent to the server as a compressed audio file. The server then issues a warning to the user and registered relatives, informing them of the potential fraud and providing information on how to access the audio data.

[0257] For example, in the case of a call that may be a phishing scam, the server will determine that the call is dangerous if certain keywords such as "bank account" or "password" are detected. Based on this, the user's device will automatically record the call and immediately notify the relevant parties. Prompts to be input into the generating AI model include "Please list keywords that indicate a phishing scam" and "Please explain the fraud detection process using natural language processing."

[0258] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0259] Step 1:

[0260] The terminal acquires user conversations in real time using an acoustic data acquisition method. The input is audio captured through the terminal's microphone. This input is acquired as a digital signal and prepared for subsequent processing.

[0261] Step 2:

[0262] The terminal converts acquired audio data into text data using automatic speech recognition (ASR) software. The input is the digital audio data obtained in step 1, which is then converted from an audio signal to text information in step 2. Specifically, the ASR engine analyzes the audio pattern and replaces it with the corresponding string. The output is the converted text data.

[0263] Step 3:

[0264] The terminal sends the converted character data to the server via the communication network. The character data from step 2 is used as input and is securely sent to the server using a network protocol (e.g., HTTPS). The output is the character data received by the server.

[0265] Step 4:

[0266] The server analyzes text data using natural language processing (NLP) techniques to assess the likelihood of fraudulent activity. The input is the text data received in step 3, and through data analysis, keywords related to fraud are identified. Specifically, the analysis algorithm tokenizes the words and scores the likelihood of fraud. The output is the evaluation result indicating the likelihood of fraudulent activity.

[0267] Step 5:

[0268] If the server determines that there is a high probability of fraudulent activity, it sends a notification to the device. The input is the evaluation result obtained in step 4, which is sent to the device as a warning message. The output is the notification displayed on the device.

[0269] Step 6:

[0270] The device automatically starts recording the conversation when it receives a notification. The input is the notification from step 5, which activates the device's recording function and saves the audio to a file. The output is the recorded audio file.

[0271] Step 7:

[0272] The terminal sends the recorded audio file to the server. The input is the audio file generated in step 6, which is then transferred to the server via the network in step 7. The output is the recorded data stored on the server.

[0273] Step 8:

[0274] The server re-analyzes the recorded audio data and notifies the user and pre-registered relatives of the results. The input is the recorded data received in step 7, and the notification is executed based on the final assessment of the likelihood of fraud. The output is a warning message sent to the user and relatives.

[0275] (Application Example 1)

[0276] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0277] In modern society, telephone-based fraud is on the rise. These methods are sophisticated, increasing the risk of many people becoming victims. Therefore, there is a need for a reliable system that can assess the possibility of fraud in real time and issue warnings quickly.

[0278] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0279] In this invention, the server includes: voice acquisition means for acquiring the user's voice; conversion means for converting the voice information acquired by the voice acquisition means into text information; analysis means for analyzing the text information obtained by the conversion means to evaluate the possibility of fraud; recording means for recording communications when the possibility of fraud is detected; notification means for notifying the user and registered family members of the possibility of fraud; and warning means for detecting the possibility of fraud in real time and sending a warning. This makes it possible to warn of fraudulent activity in advance and protect the user's information and property.

[0280] "Voice acquisition means" refers to a device or software for acquiring a user's voice as a digital signal.

[0281] "Conversion means" refers to a device or software that performs the process of converting acquired audio information into text information.

[0282] "Analysis means" refers to a device or software that utilizes natural language processing technology to evaluate the possibility of fraud based on textual information.

[0283] "Recording means" refers to a device or software that records relevant audio or text information when a potential fraud is detected.

[0284] The "notification means" is a communication means or device for notifying users and registered relatives of the possibility of fraud.

[0285] The "warning means" is a device or software for detecting the possibility of fraud in real time and immediately sending a warning to the user.

[0286] The system for realizing this invention is composed of a series of processes for acquiring, converting, analyzing, and notifying voice data. The system is implemented using the user's terminal, server, and the communication network connecting them.

[0287] The terminal functions as a voice acquisition means and acquires the user's call voice through a microphone. This voice data is converted into character information by automatic speech recognition (ASR) software in the terminal. As specific software, Google's speech recognition API is used.

[0288] The converted character information is sent to the server through the communication network. The server, as an analysis means, uses natural language processing (NLP) technology to analyze the character information. A specific natural language processing model is used to detect words and phrases related to fraud. Through this analysis, the possibility of fraud is evaluated.

[0289] When the possibility of fraud is detected, the server immediately sends a notification. As the notification means, push notifications or emails are used. The user and their relatives are notified that the corresponding communication has been recorded and there is a possibility of fraud, and details of the record access are also provided as necessary.

[0290] As a specific example, when the user receives a call containing phrases such as "credit card" or "password", the system immediately flags it as a possible fraud and notifies the registered family members that "a call with a possible fraud has been detected".

[0291] When using generative AI models, one possible prompt would be: "If FraudGuarder analyzes a suspicious call and determines it may be a scam, how will it notify the user and share the recorded call with their family?"

[0292] This allows users to receive warnings before they become victims of fraud and to take prompt action in the situation.

[0293] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0294] Step 1:

[0295] The device uses a microphone to acquire the user's call audio. The acquired analog audio signal is converted into a digital signal and input as audio data.

[0296] Step 2:

[0297] The device converts the acquired audio data into text data using automatic speech recognition (ASR) software, specifically Google's speech recognition API. The input is audio data, and the output is text information.

[0298] Step 3:

[0299] The converted character information is sent to the server via the communication network. The input is character information, and the data is sent to the server.

[0300] Step 4:

[0301] The server uses natural language processing (NLP) techniques to analyze textual information. Specifically, it uses a model to detect words and phrases related to fraud and analyzes the input textual information. The output is an evaluation result indicating the likelihood of fraud.

[0302] Step 5:

[0303] If the server determines that there is a high likelihood of fraud as a result of the analysis, it generates a warning and notifies the terminal. The input is the analysis result, and the output is sent to the user as a warning message.

[0304] Step 6:

[0305] When the terminal receives the notification from the server, it records and saves the corresponding call at the same time. The input is the notification message, and the output is the saving of the recorded data.

[0306] Step 7:

[0307] The server utilizes the notification means to inform the possibility of fraud in order to provide the user and registered family members with access rights to the recorded data. The input is the recorded data and access right information, and the output is sent to the user and family members as a notification message.

[0308] Through the above steps, the user can detect the risk of fraud in advance and take prompt action.

[0309] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion recognition model 59 and perform specific processing using the user's emotion.

[0310] The present invention provides a form in which an emotion engine for recognizing the user's emotion is combined with a system that acquires the user's call voice and analyzes the possibility of fraud. A voice acquisition means is installed on the terminal to capture the user's call in real time. The voice data is converted into text data by the conversion means and then transmitted to the server via the communication network.

[0311] The server analyzes the received text data using an analysis tool. This analysis employs a natural language processing (NLP) model to identify keywords and phrases related to fraud. Furthermore, the analysis tool works in conjunction with an emotion engine to analyze changes in the user's voice tone and pitch to determine their emotional state. The emotion engine incorporates emotional information as a corrective factor into the fraud evaluation process, thereby determining the likelihood of fraud with greater accuracy.

[0312] If a call is deemed highly likely to be fraudulent, the server will notify the device to record the call. Furthermore, if the emotion engine determines that the user's emotional state is outside the normal range, a notification will be sent immediately. This notification, including information on accessing the recording file, will be provided to the user and any pre-registered family members.

[0313] As a concrete example, consider a case where a user's voice is trembling unnaturally during a phone call. In this case, the emotion engine can detect this abnormal emotional change and generate a warning overlaid on the possibility of fraud. This warning is immediately sent to the user and their family, allowing for further action after reviewing the recorded call. This system provides users with a highly reliable means of dealing with the risk of fraud.

[0314] The following describes the processing flow.

[0315] Step 1:

[0316] The device detects when a user initiates a call, activates the voice acquisition mechanism, and captures the call audio in real time.

[0317] Step 2:

[0318] The audio data acquired by the device is converted into text data using an automatic speech recognition (ASR) system.

[0319] Step 3:

[0320] The terminal sends the converted text data to the server.

[0321] Step 4:

[0322] The server analyzes the received text data and performs keyword analysis using a natural language processing (NLP) model.

[0323] Step 5:

[0324] The server utilizes an emotion engine to analyze emotional information extracted from the voice (e.g., changes in voice tone and pitch) and determine the user's emotional state.

[0325] Step 6:

[0326] Based on the analysis results, the server comprehensively evaluates the likelihood of fraud and the emotional state, and calculates an overall score.

[0327] Step 7:

[0328] The server notifies the device to record calls that it determines are highly likely to be fraudulent or involve an abnormal emotional state.

[0329] Step 8:

[0330] The device records the specified call and sends the recording data to the server.

[0331] Step 9:

[0332] The server stores the recorded audio data and sends notifications of fraud risk or emotional disturbances to the user and pre-registered family members, and provides information on how to access the recording files.

[0333] (Example 2)

[0334] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0335] Fraudulent activities are becoming more sophisticated over time, making detection difficult with conventional methods. Furthermore, there is a need for measures to prevent attempted fraud in advance by detecting anomalies based on changes in user emotions. Conventional systems cannot interpret emotions from voice, making it difficult to accurately assess the likelihood of fraud.

[0336] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0337] In this invention, the server includes an audio acquisition means for acquiring the user's call audio, a conversion means for converting the audio data into text data, an analysis means for checking keywords and phrases related to fraud, an emotion analysis means for analyzing emotional information contained in the user's voice, an integrated evaluation means for integrating the analysis results to determine the likelihood of fraud with high accuracy, a recording means for recording the relevant call, and a notification means for issuing a notification. This enables early detection of fraud risk and prompt warning to the user.

[0338] "Voice acquisition means" refers to a device or function for collecting a user's call audio in real time.

[0339] "Conversion means" refers to the process or technology used to convert acquired audio data into text data.

[0340] "Analysis means" refers to a function that analyzes text data and identifies keywords and phrases related to fraud.

[0341] "Emotional analysis methods" refer to technologies for extracting and analyzing emotional information from user voice data.

[0342] The "integrated evaluation method" is a function that integrates analysis results and emotional information to evaluate the likelihood of fraud with high accuracy.

[0343] "Recording means" refers to technology or equipment for recording potentially fraudulent phone calls.

[0344] "Notification methods" refer to features that inform users and their registered family members of potential fraud.

[0345] The system of the present invention has the function of acquiring a user's call audio in real time and highly evaluating the possibility of fraud. When a user starts a call, the terminal captures audio data using an audio acquisition means. This audio data is converted into text data using a conversion means within the terminal. Speech recognition software (e.g., speech recognition API) is used for this conversion.

[0346] The converted text data is sent to the server via a communication network. The server analyzes the text data using a natural language processing model to identify keywords and phrases related to fraud. The server also extracts emotional information from the user's voice using sentiment analysis and uses this information to evaluate the fraud.

[0347] If a call is deemed to be at high risk of fraud, or if the user's emotional state is judged to be outside the normal range, the server will instruct the recording of the call and send a notification to the device containing information on how to access the recording data. Furthermore, this information will be notified to the user and pre-registered family members to enable a swift response.

[0348] For example, a user might receive a call from an unfamiliar caller who emphasizes urgency or speaks in an unstable tone. In this case, the server detects the tone of voice based on sentiment analysis and determines that it is likely to be a scam. The user and their family can then review the recording of the call and take appropriate action if necessary.

[0349] Example prompt for a generative AI model: "Detect changes in the user's emotions during a call and assess the risk of fraud."

[0350] This invention allows users to communicate with greater peace of mind and reduces the risk of becoming a victim of fraud.

[0351] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0352] Step 1:

[0353] When a user initiates a call, the device captures audio data in real time using an audio acquisition device. At this time, the user's call audio is the input, and audio data is generated as the output. Specifically, the device uses a microphone to acquire the audio waveform as digital data.

[0354] Step 2:

[0355] The terminal converts the acquired audio data into text data using a conversion mechanism. Here, the input is audio data, and the output is the corresponding text data. This process utilizes speech recognition software, which analyzes the pattern of the audio waveform and represents its content as text.

[0356] Step 3:

[0357] The terminal sends the converted text data to the server via the communication network. The input is text data, and the output exists as data transferred to the server. In this process, data is encrypted for security purposes to prevent unauthorized access.

[0358] Step 4:

[0359] The server processes the received text data using parsing tools to identify keywords and phrases related to fraud. The input for the parsing is the text data that reaches the server, and the output is an evaluation result indicating the likelihood of fraud. Specifically, a natural language processing model is used to detect specific words and syntax in the data.

[0360] Step 5:

[0361] The server uses emotion analysis tools in parallel with the analysis to extract emotional information from the user's voice. The input to this analysis is variables obtained from the voice data (e.g., tone, pitch), and the output is an evaluation result indicating the user's emotions. Emotion recognition software is used to analyze the voice parameters and identify changes in emotion.

[0362] Step 6:

[0363] The server integrates the analysis results and sentiment analysis results using an integrated evaluation system to determine the likelihood of fraud with high accuracy. The input to this integration is two evaluation results: analysis and sentiment, and the output is an overall fraud risk assessment value. This enables more reliable fraud detection.

[0364] Step 7:

[0365] If a call is deemed highly likely to be fraudulent, the server instructs the device to record the call and sends a notification containing information on how to access the recording data. The input is a fraud risk assessment value, and the output is notification information for the user and pre-registered family members. The server generates an access link to the recording data and includes it in the notification message to ensure rapid information dissemination.

[0366] (Application Example 2)

[0367] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0368] In recent years, with advancements in communication technology, fraudulent activities exploiting voice calls have increased. Therefore, there is a need for a system that allows users to quickly detect potential fraud during a call and take countermeasures. However, conventional fraud detection systems rely solely on keyword and phrase detection, which can result in insufficient accuracy. Furthermore, simply notifying users of potential fraud can worsen their emotional state. Therefore, a system is needed that analyzes changes in the user's emotional state and utilizes this information for fraud detection, enabling more accurate notifications.

[0369] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0370] In this invention, the server includes means for acquiring sound, means for converting it into text information, and means for analyzing emotions to determine the emotional state. This enables a more accurate assessment of the possibility of fraud, taking into account not only the possibility of fraud during a call but also the user's emotional fluctuations. This system also provides support for users to make calls with peace of mind by quickly notifying them of any abnormalities in their emotional state.

[0371] The "acoustic acquisition means" is a device that captures voice information from a user's call in real time and provides it as data necessary for subsequent processing.

[0372] A "conversion means" is a device that has the function of converting acquired audio information into text information, making it available as basic data for analysis.

[0373] An "analysis tool" is a device that analyzes textual information to assess the likelihood of fraud and makes a judgment based on specific conditions.

[0374] An "emotional analysis tool" is a device that has the function of understanding the emotional state of a user during a call and analyzing its changes to evaluate the user's mental state.

[0375] A "recording device" is a device that has the function of recording phone calls in response to the detection of potential fraud or abnormal emotional states, and saving them in a format that can be reviewed at a later date.

[0376] A "notification method" is a function that quickly informs users and registered parties of potential fraud or abnormal emotional states that have been detected.

[0377] The system implementing this invention primarily processes voice data during a call in real time and has the function of evaluating the possibility of fraud and the user's emotional state. The server converts the voice data sent from the user's terminal into text information using the Google Speech-to-Text API. This makes it possible to transform the voice information into a format that is easy to analyze.

[0378] The converted text information is sent to the server, where it is analyzed using a natural language processing model (e.g., BERT or GPT). The purpose of the analysis is to identify vocabulary and expressions related to fraud and to assess the likelihood of fraud. In addition, a sentiment analysis engine such as IBM Watson Tone Analyzer can be used to simultaneously determine the user's emotional state, and if an abnormal change in emotion is detected, the fraud likelihood assessment can be corrected.

[0379] If the server detects potential fraud or emotional instability, it will send alerts to the user and registered stakeholders using notification methods such as Firebase Cloud Messaging. This allows users to quickly recognize the situation and take appropriate action.

[0380] For example, if a user is discussing a real estate transaction over the phone and the other party speaks in an unnatural tone that seems to be pressuring them to make an immediate decision, the system will detect this and analyze the emotional changes that indicate the user's anxiety is increasing. Based on this information, a potential fraud alert will be sent immediately.

[0381] An example of a prompt message is: "Assess the likelihood of fraud based on the user's call content. Also, analyze the user's emotional changes during the call and provide details if anything seems unnatural."

[0382] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0383] Step 1:

[0384] The terminal captures the user's call audio using an acoustic acquisition device. The input is raw audio data, and the output is real-time captured audio data. Specifically, the terminal's microphone collects the audio and converts it into a digital signal.

[0385] Step 2:

[0386] The device converts captured audio data into text information using a conversion mechanism. The input is audio data, and the output is text data. Specifically, the Google Speech-to-Text API is used to analyze the audio data and generate the corresponding text.

[0387] Step 3:

[0388] The server uses analysis tools to evaluate the likelihood of fraud in text data sent from a terminal. The input is converted character information, and the output is the evaluation result regarding the likelihood of fraud. Specifically, a natural language processing model (such as BERT or GPT) analyzes the text and detects vocabulary and expressions related to fraud.

[0389] Step 4:

[0390] The server analyzes the text information and uses sentiment analysis tools to determine the user's emotional state. The input is text data, and the output is the evaluation result of the emotional state. Specifically, IBM Watson Tone Analyzer detects emotional fluctuations from the text and analyzes the results.

[0391] Step 5:

[0392] The server integrates the fraud likelihood assessment and the emotional state assessment to make a final decision. The input is the output results from steps 3 and 4, and the output is the final fraud likelihood assessment and the necessary actions. Specifically, a fraud warning alert is generated based on these two evaluation results.

[0393] Step 6:

[0394] The server uses notification methods to send alerts to users and registered stakeholders. The input is the final fraud warning alert, and the output is the notification sent to users and stakeholders. Specifically, Firebase Cloud Messaging is used to share information by displaying the alert on stakeholders' devices immediately.

[0395] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0396] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0397] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0398] [Third Embodiment]

[0399] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0400] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0401] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0402] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0403] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0404] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0405] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0406] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0407] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0408] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0409] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0410] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0411] For the implementation of this invention, a user terminal, a server, and a communication network connecting them are essential. In this system configuration, the terminal is equipped with voice acquisition means to acquire voice during a call in real time. The acquired voice data is converted into text data using automatic speech recognition (ASR) software on the terminal.

[0412] The converted text data is sent to the server via a communication network. The server uses natural language processing (NLP) techniques to analyze this text data and assess the likelihood of fraud. The analysis employs a model that meticulously searches for keywords and phrases related to fraud.

[0413] If a call is deemed potentially fraudulent, the server sends a notification to the device, and the device automatically records the call. The recorded audio data is then sent to the server, which sends a notification to the user and any pre-registered family members. This notification not only informs them of the potential fraud but also provides them with access to the recorded audio.

[0414] As a concrete example, consider a scenario where a user receives a phishing call. If the call contains suspicious keywords such as "bank account" or "password," the server immediately determines that it is highly likely to be a scam. The receiving device records the call, and a warning is sent to the user and their family via the server. This process allows the user and their family to identify the risk of fraud and take prompt action.

[0415] The following describes the processing flow.

[0416] Step 1:

[0417] The terminal detects when a user initiates a call and activates the voice acquisition mechanism to capture the call audio in real time.

[0418] Step 2:

[0419] The device transmits the acquired audio data to the Automatic Speech Recognition (ASR) system in real time, converting the audio into text data.

[0420] Step 3:

[0421] The terminal utilizes a communication module to send the converted text data to the server, transferring the text data to the server.

[0422] Step 4:

[0423] The server uses natural language processing (NLP) techniques to analyze the received text data and runs a model designed to assess whether it is potentially fraudulent.

[0424] Step 5:

[0425] If the server determines, based on its analysis, that there is a high probability of fraud, it generates a fraud alert and sends that information to the device.

[0426] Step 6:

[0427] The device receives a notification from the server, automatically records the relevant call, and generates a recording file.

[0428] Step 7:

[0429] The device transfers the recorded audio data to the server, and the server stores the recording file.

[0430] Step 8:

[0431] The server sends a notification to the user and their pre-registered family members, containing information about the potential for fraud and how to access the recording files.

[0432] (Example 1)

[0433] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0434] In modern communication, users are at increasing risk of becoming involved in fraudulent calls related to malicious activity. Because there is a lack of means to identify such calls in real time and effectively warn users, preventing fraud is difficult.

[0435] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0436] In this invention, the server includes means for acquiring acoustic data to capture user conversations, means for converting the acoustic data acquired by the acoustic data acquisition means into text data, and means for analyzing the converted text data to evaluate the possibility of fraudulent activity. This makes it possible to quickly identify calls related to fraudulent activity and to issue appropriate warnings to the user and their relatives.

[0437] "Acoustic data acquisition means" refers to a device or technology for acquiring user conversations in real time, and mainly includes microphones and related signal processing technologies.

[0438] "Means of converting into text data" refers to technologies that convert acquired audio data so that it can be treated as text information, and automatic speech recognition (ASR) technology falls under this category.

[0439] "Means of analysis" refers to techniques for analyzing converted character data and evaluating the likelihood of fraudulent activity, including using natural language processing (NLP) techniques to identify words and expressions related to fraud.

[0440] "Means of recording" means a device or process for saving a conversation in which potential misconduct has been detected, and includes techniques for electronically recording audio data.

[0441] "Means of notification" refers to methods of issuing warnings to users and registered relatives in the event of potential fraudulent activity, such as email or in-app notifications.

[0442] To implement this invention, a user's terminal, a server, and a communication network connecting them are required. The terminal is equipped with acoustic data acquisition means to acquire the user's conversation in real time. A microphone is used for this acquisition, and acoustic sensing technology built into smartphones and tablets is applied.

[0443] The device converts the acquired audio data into text data using automatic speech recognition (ASR) software. Commonly used software for this purpose includes speech-to-text technologies, such as Google Speech-to-Text and Watson Speech-to-Text from Google Inc.

[0444] The terminal sends the converted character data to the server via the communication network. The server analyzes this character data using natural language processing (NLP) techniques to evaluate the likelihood of fraudulent activity. Python's natural language processing libraries, such as NLTK and spaCy, are used for the analysis. The server tokenizes the information and identifies relevant keywords to numerically analyze the likelihood of fraud.

[0445] When the server detects potential fraudulent activity, it notifies the terminal accordingly. Upon receiving the notification, the terminal immediately begins recording the conversation and generates audio data. This audio data is sent to the server as a compressed audio file. The server then issues a warning to the user and registered relatives, informing them of the potential fraud and providing information on how to access the audio data.

[0446] For example, in the case of a call that may be a phishing scam, the server will determine that the call is dangerous if certain keywords such as "bank account" or "password" are detected. Based on this, the user's device will automatically record the call and immediately notify the relevant parties. Prompts to be input into the generating AI model include "Please list keywords that indicate a phishing scam" and "Please explain the fraud detection process using natural language processing."

[0447] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0448] Step 1:

[0449] The terminal acquires user conversations in real time using an acoustic data acquisition method. The input is audio captured through the terminal's microphone. This input is acquired as a digital signal and prepared for subsequent processing.

[0450] Step 2:

[0451] The terminal converts acquired audio data into text data using automatic speech recognition (ASR) software. The input is the digital audio data obtained in step 1, which is then converted from an audio signal to text information in step 2. Specifically, the ASR engine analyzes the audio pattern and replaces it with the corresponding string. The output is the converted text data.

[0452] Step 3:

[0453] The terminal sends the converted character data to the server via the communication network. The character data from step 2 is used as input and is securely sent to the server using a network protocol (e.g., HTTPS). The output is the character data received by the server.

[0454] Step 4:

[0455] The server analyzes text data using natural language processing (NLP) techniques to assess the likelihood of fraudulent activity. The input is the text data received in step 3, and through data analysis, keywords related to fraud are identified. Specifically, the analysis algorithm tokenizes the words and scores the likelihood of fraud. The output is the evaluation result indicating the likelihood of fraudulent activity.

[0456] Step 5:

[0457] If the server determines that there is a high probability of fraudulent activity, it sends a notification to the device. The input is the evaluation result obtained in step 4, which is sent to the device as a warning message. The output is the notification displayed on the device.

[0458] Step 6:

[0459] The device automatically starts recording the conversation when it receives a notification. The input is the notification from step 5, which activates the device's recording function and saves the audio to a file. The output is the recorded audio file.

[0460] Step 7:

[0461] The terminal sends the recorded audio file to the server. The input is the audio file generated in step 6, which is then transferred to the server via the network in step 7. The output is the recorded data stored on the server.

[0462] Step 8:

[0463] The server re-analyzes the recorded audio data and notifies the user and pre-registered relatives of the results. The input is the recorded data received in step 7, and the notification is executed based on the final assessment of the likelihood of fraud. The output is a warning message sent to the user and relatives.

[0464] (Application Example 1)

[0465] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0466] In modern society, telephone-based fraud is on the rise. These methods are sophisticated, increasing the risk of many people becoming victims. Therefore, there is a need for a reliable system that can assess the possibility of fraud in real time and issue warnings quickly.

[0467] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0468] In this invention, the server includes: voice acquisition means for acquiring the user's voice; conversion means for converting the voice information acquired by the voice acquisition means into text information; analysis means for analyzing the text information obtained by the conversion means to evaluate the possibility of fraud; recording means for recording communications when the possibility of fraud is detected; notification means for notifying the user and registered family members of the possibility of fraud; and warning means for detecting the possibility of fraud in real time and sending a warning. This makes it possible to warn of fraudulent activity in advance and protect the user's information and property.

[0469] "Voice acquisition means" refers to a device or software for acquiring a user's voice as a digital signal.

[0470] "Conversion means" refers to a device or software that performs the process of converting acquired audio information into text information.

[0471] "Analysis means" refers to a device or software that utilizes natural language processing technology to evaluate the possibility of fraud based on textual information.

[0472] "Recording means" refers to a device or software that records relevant audio or text information when a potential fraud is detected.

[0473] "Notification means" refers to a means or device for communicating with users and their registered family members to inform them of the possibility of fraud.

[0474] A "warning device" is a device or software that detects potential fraud in real time and immediately issues a warning to the user.

[0475] The system for realizing this invention consists of a series of processes for acquiring, converting, analyzing, and notifying voice data. The system is implemented using a user's terminal, a server, and a communication network connecting them.

[0476] The device functions as a means of voice acquisition, capturing the user's call audio via its microphone. This audio data is converted into text by the device's built-in automatic speech recognition (ASR) software. Specifically, Google's speech recognition API is used.

[0477] The converted text information is transmitted to the server via a communication network. The server analyzes the text information using natural language processing (NLP) techniques. Specific NLP models are used to detect words and phrases related to fraudulent activity. This analysis assesses the likelihood of fraud.

[0478] If a potential scam is detected, the server will immediately issue a notification. Notification methods include push notifications and email. The user and their family members will be notified that the relevant communication has been recorded and that it may be fraudulent, and details of access to the recording will be provided if necessary.

[0479] For example, if a user receives a call containing phrases such as "credit card" or "password," the system immediately flags it as potentially fraudulent and notifies registered family members that "a potentially fraudulent call has been detected."

[0480] When using generative AI models, one possible prompt would be: "If FraudGuarder analyzes a suspicious call and determines it may be a scam, how will it notify the user and share the recorded call with their family?"

[0481] This allows users to receive warnings before they become victims of fraud and to take prompt action in the situation.

[0482] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0483] Step 1:

[0484] The device uses a microphone to acquire the user's call audio. The acquired analog audio signal is converted into a digital signal and input as audio data.

[0485] Step 2:

[0486] The device converts the acquired audio data into text data using automatic speech recognition (ASR) software, specifically Google's speech recognition API. The input is audio data, and the output is text information.

[0487] Step 3:

[0488] The converted character information is sent to the server via the communication network. The input is character information, and the data is sent to the server.

[0489] Step 4:

[0490] The server uses natural language processing (NLP) techniques to analyze textual information. Specifically, it uses a model to detect words and phrases related to fraud and analyzes the input textual information. The output is an evaluation result indicating the likelihood of fraud.

[0491] Step 5:

[0492] If the server determines, based on the analysis, that there is a high probability of fraud, it generates a warning and notifies the terminal. The input is the analysis result, and the output is sent to the user as a warning message.

[0493] Step 6:

[0494] The device records and saves the corresponding call upon receiving a notification from the server. The input is the notification message, and the output is the saving of the recorded data.

[0495] Step 7:

[0496] The server uses notification mechanisms to alert users and registered family members to potential fraud, granting them access to the recorded data. Inputs are the recorded data and access rights information, while output is sent to users and family members as notification messages.

[0497] By following these steps, users can detect the risk of fraud in advance and take prompt action.

[0498] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0499] This invention provides a system that combines a user's call audio acquisition and fraud analysis with an emotion engine that recognizes the user's emotions. A voice acquisition means is installed on the terminal to capture the user's call in real time. The voice data is converted into text data by a conversion means and then transmitted to a server via a communication network.

[0500] The server analyzes the received text data using an analysis tool. This analysis employs a natural language processing (NLP) model to identify keywords and phrases related to fraud. Furthermore, the analysis tool works in conjunction with an emotion engine to analyze changes in the user's voice tone and pitch to determine their emotional state. The emotion engine incorporates emotional information as a corrective factor into the fraud evaluation process, thereby determining the likelihood of fraud with greater accuracy.

[0501] If a call is deemed highly likely to be fraudulent, the server will notify the device to record the call. Furthermore, if the emotion engine determines that the user's emotional state is outside the normal range, a notification will be sent immediately. This notification, including information on accessing the recording file, will be provided to the user and any pre-registered family members.

[0502] As a concrete example, consider a case where a user's voice is trembling unnaturally during a phone call. In this case, the emotion engine can detect this abnormal emotional change and generate a warning overlaid on the possibility of fraud. This warning is immediately sent to the user and their family, allowing for further action after reviewing the recorded call. This system provides users with a highly reliable means of dealing with the risk of fraud.

[0503] The following describes the processing flow.

[0504] Step 1:

[0505] The device detects when a user initiates a call, activates the voice acquisition mechanism, and captures the call audio in real time.

[0506] Step 2:

[0507] The audio data acquired by the device is converted into text data using an automatic speech recognition (ASR) system.

[0508] Step 3:

[0509] The terminal sends the converted text data to the server.

[0510] Step 4:

[0511] The server analyzes the received text data and performs keyword analysis using a natural language processing (NLP) model.

[0512] Step 5:

[0513] The server utilizes an emotion engine to analyze emotional information extracted from the voice (e.g., changes in voice tone and pitch) and determine the user's emotional state.

[0514] Step 6:

[0515] Based on the analysis results, the server comprehensively evaluates the likelihood of fraud and the emotional state, and calculates an overall score.

[0516] Step 7:

[0517] The server notifies the device to record calls that it determines are highly likely to be fraudulent or involve an abnormal emotional state.

[0518] Step 8:

[0519] The device records the specified call and sends the recording data to the server.

[0520] Step 9:

[0521] The server stores the recorded audio data and sends notifications of fraud risk or emotional disturbances to the user and pre-registered family members, and provides information on how to access the recording files.

[0522] (Example 2)

[0523] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0524] Fraudulent activities are becoming more sophisticated over time, making detection difficult with conventional methods. Furthermore, there is a need for measures to prevent attempted fraud in advance by detecting anomalies based on changes in user emotions. Conventional systems cannot interpret emotions from voice, making it difficult to accurately assess the likelihood of fraud.

[0525] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0526] In this invention, the server includes an audio acquisition means for acquiring the user's call audio, a conversion means for converting the audio data into text data, an analysis means for checking keywords and phrases related to fraud, an emotion analysis means for analyzing emotional information contained in the user's voice, an integrated evaluation means for integrating the analysis results to determine the likelihood of fraud with high accuracy, a recording means for recording the relevant call, and a notification means for issuing a notification. This enables early detection of fraud risk and prompt warning to the user.

[0527] "Voice acquisition means" refers to a device or function for collecting a user's call audio in real time.

[0528] "Conversion means" refers to the process or technology used to convert acquired audio data into text data.

[0529] "Analysis means" refers to a function that analyzes text data and identifies keywords and phrases related to fraud.

[0530] "Emotional analysis methods" refer to technologies for extracting and analyzing emotional information from user voice data.

[0531] The "integrated evaluation method" is a function that integrates analysis results and emotional information to evaluate the likelihood of fraud with high accuracy.

[0532] "Recording means" refers to technology or equipment for recording potentially fraudulent phone calls.

[0533] "Notification methods" refer to features that inform users and their registered family members of potential fraud.

[0534] The system of the present invention has the function of acquiring a user's call audio in real time and highly evaluating the possibility of fraud. When a user starts a call, the terminal captures audio data using an audio acquisition means. This audio data is converted into text data using a conversion means within the terminal. Speech recognition software (e.g., speech recognition API) is used for this conversion.

[0535] The converted text data is sent to the server via a communication network. The server analyzes the text data using a natural language processing model to identify keywords and phrases related to fraud. The server also extracts emotional information from the user's voice using sentiment analysis and uses this information to evaluate the fraud.

[0536] If a call is deemed to be at high risk of fraud, or if the user's emotional state is judged to be outside the normal range, the server will instruct the recording of the call and send a notification to the device containing information on how to access the recording data. Furthermore, this information will be notified to the user and pre-registered family members to enable a swift response.

[0537] For example, a user might receive a call from an unfamiliar caller who emphasizes urgency or speaks in an unstable tone. In this case, the server detects the tone of voice based on sentiment analysis and determines that it is likely to be a scam. The user and their family can then review the recording of the call and take appropriate action if necessary.

[0538] Example prompt for a generative AI model: "Detect changes in the user's emotions during a call and assess the risk of fraud."

[0539] This invention allows users to communicate with greater peace of mind and reduces the risk of becoming a victim of fraud.

[0540] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0541] Step 1:

[0542] When a user initiates a call, the device captures audio data in real time using an audio acquisition device. At this time, the user's call audio is the input, and audio data is generated as the output. Specifically, the device uses a microphone to acquire the audio waveform as digital data.

[0543] Step 2:

[0544] The terminal converts the acquired audio data into text data using a conversion mechanism. Here, the input is audio data, and the output is the corresponding text data. This process utilizes speech recognition software, which analyzes the pattern of the audio waveform and represents its content as text.

[0545] Step 3:

[0546] The terminal sends the converted text data to the server via the communication network. The input is text data, and the output exists as data transferred to the server. In this process, data is encrypted for security purposes to prevent unauthorized access.

[0547] Step 4:

[0548] The server processes the received text data using parsing tools to identify keywords and phrases related to fraud. The input for the parsing is the text data that reaches the server, and the output is an evaluation result indicating the likelihood of fraud. Specifically, a natural language processing model is used to detect specific words and syntax in the data.

[0549] Step 5:

[0550] The server uses emotion analysis tools in parallel with the analysis to extract emotional information from the user's voice. The input to this analysis is variables obtained from the voice data (e.g., tone, pitch), and the output is an evaluation result indicating the user's emotions. Emotion recognition software is used to analyze the voice parameters and identify changes in emotion.

[0551] Step 6:

[0552] The server integrates the analysis results and sentiment analysis results using an integrated evaluation system to determine the likelihood of fraud with high accuracy. The input to this integration is two evaluation results: analysis and sentiment, and the output is an overall fraud risk assessment value. This enables more reliable fraud detection.

[0553] Step 7:

[0554] If a call is deemed highly likely to be fraudulent, the server instructs the device to record the call and sends a notification containing information on how to access the recording data. The input is a fraud risk assessment value, and the output is notification information for the user and pre-registered family members. The server generates an access link to the recording data and includes it in the notification message to ensure rapid information dissemination.

[0555] (Application Example 2)

[0556] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0557] In recent years, with advancements in communication technology, fraudulent activities exploiting voice calls have increased. Therefore, there is a need for a system that allows users to quickly detect potential fraud during a call and take countermeasures. However, conventional fraud detection systems rely solely on keyword and phrase detection, which can result in insufficient accuracy. Furthermore, simply notifying users of potential fraud can worsen their emotional state. Therefore, a system is needed that analyzes changes in the user's emotional state and utilizes this information for fraud detection, enabling more accurate notifications.

[0558] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0559] In this invention, the server includes means for acquiring sound, means for converting it into text information, and means for analyzing emotions to determine the emotional state. This enables a more accurate assessment of the possibility of fraud, taking into account not only the possibility of fraud during a call but also the user's emotional fluctuations. This system also provides support for users to make calls with peace of mind by quickly notifying them of any abnormalities in their emotional state.

[0560] The "acoustic acquisition means" is a device that captures voice information from a user's call in real time and provides it as data necessary for subsequent processing.

[0561] A "conversion means" is a device that has the function of converting acquired audio information into text information, making it available as basic data for analysis.

[0562] An "analysis tool" is a device that analyzes textual information to assess the likelihood of fraud and makes a judgment based on specific conditions.

[0563] An "emotional analysis tool" is a device that has the function of understanding the emotional state of a user during a call and analyzing its changes to evaluate the user's mental state.

[0564] A "recording device" is a device that has the function of recording phone calls in response to the detection of potential fraud or abnormal emotional states, and saving them in a format that can be reviewed at a later date.

[0565] A "notification method" is a function that quickly informs users and registered parties of potential fraud or abnormal emotional states that have been detected.

[0566] The system implementing this invention primarily processes voice data during a call in real time and has the function of evaluating the possibility of fraud and the user's emotional state. The server converts the voice data sent from the user's terminal into text information using the Google Speech-to-Text API. This makes it possible to transform the voice information into a format that is easy to analyze.

[0567] The converted text information is sent to the server, where it is analyzed using a natural language processing model (e.g., BERT or GPT). The purpose of the analysis is to identify vocabulary and expressions related to fraud and to assess the likelihood of fraud. In addition, a sentiment analysis engine such as IBM Watson Tone Analyzer can be used to simultaneously determine the user's emotional state, and if an abnormal change in emotion is detected, the fraud likelihood assessment can be corrected.

[0568] If the server detects potential fraud or emotional instability, it will send alerts to the user and registered stakeholders using notification methods such as Firebase Cloud Messaging. This allows users to quickly recognize the situation and take appropriate action.

[0569] For example, if a user is discussing a real estate transaction over the phone and the other party speaks in an unnatural tone that seems to be pressuring them to make an immediate decision, the system will detect this and analyze the emotional changes that indicate the user's anxiety is increasing. Based on this information, a potential fraud alert will be sent immediately.

[0570] An example of a prompt message is: "Assess the likelihood of fraud based on the user's call content. Also, analyze the user's emotional changes during the call and provide details if anything seems unnatural."

[0571] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0572] Step 1:

[0573] The terminal captures the user's call audio using an acoustic acquisition device. The input is raw audio data, and the output is real-time captured audio data. Specifically, the terminal's microphone collects the audio and converts it into a digital signal.

[0574] Step 2:

[0575] The device converts captured audio data into text information using a conversion mechanism. The input is audio data, and the output is text data. Specifically, the Google Speech-to-Text API is used to analyze the audio data and generate the corresponding text.

[0576] Step 3:

[0577] The server uses analysis tools to evaluate the likelihood of fraud in text data sent from a terminal. The input is converted character information, and the output is the evaluation result regarding the likelihood of fraud. Specifically, a natural language processing model (such as BERT or GPT) analyzes the text and detects vocabulary and expressions related to fraud.

[0578] Step 4:

[0579] The server analyzes the text information and uses sentiment analysis tools to determine the user's emotional state. The input is text data, and the output is the evaluation result of the emotional state. Specifically, IBM Watson Tone Analyzer detects emotional fluctuations from the text and analyzes the results.

[0580] Step 5:

[0581] The server integrates the fraud likelihood assessment and the emotional state assessment to make a final decision. The input is the output results from steps 3 and 4, and the output is the final fraud likelihood assessment and the necessary actions. Specifically, a fraud warning alert is generated based on these two evaluation results.

[0582] Step 6:

[0583] The server uses notification methods to send alerts to users and registered stakeholders. The input is the final fraud warning alert, and the output is the notification sent to users and stakeholders. Specifically, Firebase Cloud Messaging is used to share information by displaying the alert on stakeholders' devices immediately.

[0584] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0585] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0586] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0587] [Fourth Embodiment]

[0588] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0589] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0590] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0591] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0592] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0593] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0594] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0595] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0596] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0597] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0598] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0599] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0600] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0601] For the implementation of this invention, a user terminal, a server, and a communication network connecting them are essential. In this system configuration, the terminal is equipped with voice acquisition means to acquire voice during a call in real time. The acquired voice data is converted into text data using automatic speech recognition (ASR) software on the terminal.

[0602] The converted text data is sent to the server via a communication network. The server uses natural language processing (NLP) techniques to analyze this text data and assess the likelihood of fraud. The analysis employs a model that meticulously searches for keywords and phrases related to fraud.

[0603] If a call is deemed potentially fraudulent, the server sends a notification to the device, and the device automatically records the call. The recorded audio data is then sent to the server, which sends a notification to the user and any pre-registered family members. This notification not only informs them of the potential fraud but also provides them with access to the recorded audio.

[0604] As a concrete example, consider a scenario where a user receives a phishing call. If the call contains suspicious keywords such as "bank account" or "password," the server immediately determines that it is highly likely to be a scam. The receiving device records the call, and a warning is sent to the user and their family via the server. This process allows the user and their family to identify the risk of fraud and take prompt action.

[0605] The following describes the processing flow.

[0606] Step 1:

[0607] The terminal detects when a user initiates a call and activates the voice acquisition mechanism to capture the call audio in real time.

[0608] Step 2:

[0609] The device transmits the acquired audio data to the Automatic Speech Recognition (ASR) system in real time, converting the audio into text data.

[0610] Step 3:

[0611] The terminal utilizes a communication module to send the converted text data to the server, transferring the text data to the server.

[0612] Step 4:

[0613] The server uses natural language processing (NLP) techniques to analyze the received text data and runs a model designed to assess whether it is potentially fraudulent.

[0614] Step 5:

[0615] If the server determines, based on its analysis, that there is a high probability of fraud, it generates a fraud alert and sends that information to the device.

[0616] Step 6:

[0617] The device receives a notification from the server, automatically records the relevant call, and generates a recording file.

[0618] Step 7:

[0619] The device transfers the recorded audio data to the server, and the server stores the recording file.

[0620] Step 8:

[0621] The server sends a notification to the user and their pre-registered family members, containing information about the potential for fraud and how to access the recording files.

[0622] (Example 1)

[0623] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0624] In modern communication, users are at increasing risk of becoming involved in fraudulent calls related to malicious activity. Because there is a lack of means to identify such calls in real time and effectively warn users, preventing fraud is difficult.

[0625] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0626] In this invention, the server includes means for acquiring acoustic data to capture user conversations, means for converting the acoustic data acquired by the acoustic data acquisition means into text data, and means for analyzing the converted text data to evaluate the possibility of fraudulent activity. This makes it possible to quickly identify calls related to fraudulent activity and to issue appropriate warnings to the user and their relatives.

[0627] "Acoustic data acquisition means" refers to a device or technology for acquiring user conversations in real time, and mainly includes microphones and related signal processing technologies.

[0628] "Means of converting into text data" refers to technologies that convert acquired audio data so that it can be treated as text information, and automatic speech recognition (ASR) technology falls under this category.

[0629] "Means of analysis" refers to techniques for analyzing converted character data and evaluating the likelihood of fraudulent activity, including using natural language processing (NLP) techniques to identify words and expressions related to fraud.

[0630] "Means of recording" means a device or process for saving a conversation in which potential misconduct has been detected, and includes techniques for electronically recording audio data.

[0631] "Means of notification" refers to methods of issuing warnings to users and registered relatives in the event of potential fraudulent activity, such as email or in-app notifications.

[0632] To implement this invention, a user's terminal, a server, and a communication network connecting them are required. The terminal is equipped with acoustic data acquisition means to acquire the user's conversation in real time. A microphone is used for this acquisition, and acoustic sensing technology built into smartphones and tablets is applied.

[0633] The device converts the acquired audio data into text data using automatic speech recognition (ASR) software. Commonly used software for this purpose includes speech-to-text technologies, such as Google Speech-to-Text and Watson Speech-to-Text from Google Inc.

[0634] The terminal sends the converted character data to the server via the communication network. The server analyzes this character data using natural language processing (NLP) techniques to evaluate the likelihood of fraudulent activity. Python's natural language processing libraries, such as NLTK and spaCy, are used for the analysis. The server tokenizes the information and identifies relevant keywords to numerically analyze the likelihood of fraud.

[0635] When the server detects potential fraudulent activity, it notifies the terminal accordingly. Upon receiving the notification, the terminal immediately begins recording the conversation and generates audio data. This audio data is sent to the server as a compressed audio file. The server then issues a warning to the user and registered relatives, informing them of the potential fraud and providing information on how to access the audio data.

[0636] For example, in the case of a call that may be a phishing scam, the server will determine that the call is dangerous if certain keywords such as "bank account" or "password" are detected. Based on this, the user's device will automatically record the call and immediately notify the relevant parties. Prompts to be input into the generating AI model include "Please list keywords that indicate a phishing scam" and "Please explain the fraud detection process using natural language processing."

[0637] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0638] Step 1:

[0639] The terminal acquires user conversations in real time using an acoustic data acquisition method. The input is audio captured through the terminal's microphone. This input is acquired as a digital signal and prepared for subsequent processing.

[0640] Step 2:

[0641] The terminal converts acquired audio data into text data using automatic speech recognition (ASR) software. The input is the digital audio data obtained in step 1, which is then converted from an audio signal to text information in step 2. Specifically, the ASR engine analyzes the audio pattern and replaces it with the corresponding string. The output is the converted text data.

[0642] Step 3:

[0643] The terminal sends the converted character data to the server via the communication network. The character data from step 2 is used as input and is securely sent to the server using a network protocol (e.g., HTTPS). The output is the character data received by the server.

[0644] Step 4:

[0645] The server analyzes text data using natural language processing (NLP) techniques to assess the likelihood of fraudulent activity. The input is the text data received in step 3, and through data analysis, keywords related to fraud are identified. Specifically, the analysis algorithm tokenizes the words and scores the likelihood of fraud. The output is the evaluation result indicating the likelihood of fraudulent activity.

[0646] Step 5:

[0647] If the server determines that there is a high probability of fraudulent activity, it sends a notification to the device. The input is the evaluation result obtained in step 4, which is sent to the device as a warning message. The output is the notification displayed on the device.

[0648] Step 6:

[0649] The device automatically starts recording the conversation when it receives a notification. The input is the notification from step 5, which activates the device's recording function and saves the audio to a file. The output is the recorded audio file.

[0650] Step 7:

[0651] The terminal sends the recorded audio file to the server. The input is the audio file generated in step 6, which is then transferred to the server via the network in step 7. The output is the recorded data stored on the server.

[0652] Step 8:

[0653] The server re-analyzes the recorded audio data and notifies the user and pre-registered relatives of the results. The input is the recorded data received in step 7, and the notification is executed based on the final assessment of the likelihood of fraud. The output is a warning message sent to the user and relatives.

[0654] (Application Example 1)

[0655] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0656] In modern society, telephone-based fraud is on the rise. These methods are sophisticated, increasing the risk of many people becoming victims. Therefore, there is a need for a reliable system that can assess the possibility of fraud in real time and issue warnings quickly.

[0657] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0658] In this invention, the server includes: voice acquisition means for acquiring the user's voice; conversion means for converting the voice information acquired by the voice acquisition means into text information; analysis means for analyzing the text information obtained by the conversion means to evaluate the possibility of fraud; recording means for recording communications when the possibility of fraud is detected; notification means for notifying the user and registered family members of the possibility of fraud; and warning means for detecting the possibility of fraud in real time and sending a warning. This makes it possible to warn of fraudulent activity in advance and protect the user's information and property.

[0659] "Voice acquisition means" refers to a device or software for acquiring a user's voice as a digital signal.

[0660] "Conversion means" refers to a device or software that performs the process of converting acquired audio information into text information.

[0661] "Analysis means" refers to a device or software that utilizes natural language processing technology to evaluate the possibility of fraud based on textual information.

[0662] "Recording means" refers to a device or software that records relevant audio or text information when a potential fraud is detected.

[0663] "Notification means" refers to a means or device for communicating with users and their registered family members to inform them of the possibility of fraud.

[0664] A "warning device" is a device or software that detects potential fraud in real time and immediately issues a warning to the user.

[0665] The system for realizing this invention consists of a series of processes for acquiring, converting, analyzing, and notifying voice data. The system is implemented using a user's terminal, a server, and a communication network connecting them.

[0666] The device functions as a means of voice acquisition, capturing the user's call audio via its microphone. This audio data is converted into text by the device's built-in automatic speech recognition (ASR) software. Specifically, Google's speech recognition API is used.

[0667] The converted text information is transmitted to the server via a communication network. The server analyzes the text information using natural language processing (NLP) techniques. Specific NLP models are used to detect words and phrases related to fraudulent activity. This analysis assesses the likelihood of fraud.

[0668] If a potential scam is detected, the server will immediately issue a notification. Notification methods include push notifications and email. The user and their family members will be notified that the relevant communication has been recorded and that it may be fraudulent, and details of access to the recording will be provided if necessary.

[0669] For example, if a user receives a call containing phrases such as "credit card" or "password," the system immediately flags it as potentially fraudulent and notifies registered family members that "a potentially fraudulent call has been detected."

[0670] When using generative AI models, one possible prompt would be: "If FraudGuarder analyzes a suspicious call and determines it may be a scam, how will it notify the user and share the recorded call with their family?"

[0671] This allows users to receive warnings before they become victims of fraud and to take prompt action in the situation.

[0672] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0673] Step 1:

[0674] The device uses a microphone to acquire the user's call audio. The acquired analog audio signal is converted into a digital signal and input as audio data.

[0675] Step 2:

[0676] The device converts the acquired audio data into text data using automatic speech recognition (ASR) software, specifically Google's speech recognition API. The input is audio data, and the output is text information.

[0677] Step 3:

[0678] The converted character information is sent to the server via the communication network. The input is character information, and the data is sent to the server.

[0679] Step 4:

[0680] The server uses natural language processing (NLP) techniques to analyze textual information. Specifically, it uses a model to detect words and phrases related to fraud and analyzes the input textual information. The output is an evaluation result indicating the likelihood of fraud.

[0681] Step 5:

[0682] If the server determines, based on the analysis, that there is a high probability of fraud, it generates a warning and notifies the terminal. The input is the analysis result, and the output is sent to the user as a warning message.

[0683] Step 6:

[0684] The device records and saves the corresponding call upon receiving a notification from the server. The input is the notification message, and the output is the saving of the recorded data.

[0685] Step 7:

[0686] The server uses notification mechanisms to alert users and registered family members to potential fraud, granting them access to the recorded data. Inputs are the recorded data and access rights information, while output is sent to users and family members as notification messages.

[0687] By following these steps, users can detect the risk of fraud in advance and take prompt action.

[0688] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0689] This invention provides a system that combines a user's call audio acquisition and fraud analysis with an emotion engine that recognizes the user's emotions. A voice acquisition means is installed on the terminal to capture the user's call in real time. The voice data is converted into text data by a conversion means and then transmitted to a server via a communication network.

[0690] The server analyzes the received text data using an analysis tool. This analysis employs a natural language processing (NLP) model to identify keywords and phrases related to fraud. Furthermore, the analysis tool works in conjunction with an emotion engine to analyze changes in the user's voice tone and pitch to determine their emotional state. The emotion engine incorporates emotional information as a corrective factor into the fraud evaluation process, thereby determining the likelihood of fraud with greater accuracy.

[0691] If a call is deemed highly likely to be fraudulent, the server will notify the device to record the call. Furthermore, if the emotion engine determines that the user's emotional state is outside the normal range, a notification will be sent immediately. This notification, including information on accessing the recording file, will be provided to the user and any pre-registered family members.

[0692] As a concrete example, consider a case where a user's voice is trembling unnaturally during a phone call. In this case, the emotion engine can detect this abnormal emotional change and generate a warning overlaid on the possibility of fraud. This warning is immediately sent to the user and their family, allowing for further action after reviewing the recorded call. This system provides users with a highly reliable means of dealing with the risk of fraud.

[0693] The following describes the processing flow.

[0694] Step 1:

[0695] The device detects when a user initiates a call, activates the voice acquisition mechanism, and captures the call audio in real time.

[0696] Step 2:

[0697] The audio data acquired by the device is converted into text data using an automatic speech recognition (ASR) system.

[0698] Step 3:

[0699] The terminal sends the converted text data to the server.

[0700] Step 4:

[0701] The server analyzes the received text data and performs keyword analysis using a natural language processing (NLP) model.

[0702] Step 5:

[0703] The server utilizes an emotion engine to analyze emotional information extracted from the voice (e.g., changes in voice tone and pitch) and determine the user's emotional state.

[0704] Step 6:

[0705] Based on the analysis results, the server comprehensively evaluates the likelihood of fraud and the emotional state, and calculates an overall score.

[0706] Step 7:

[0707] The server notifies the device to record calls that it determines are highly likely to be fraudulent or involve an abnormal emotional state.

[0708] Step 8:

[0709] The device records the specified call and sends the recording data to the server.

[0710] Step 9:

[0711] The server stores the recorded audio data and sends notifications of fraud risk or emotional disturbances to the user and pre-registered family members, and provides information on how to access the recording files.

[0712] (Example 2)

[0713] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0714] Fraudulent activities are becoming more sophisticated over time, making detection difficult with conventional methods. Furthermore, there is a need for measures to prevent attempted fraud in advance by detecting anomalies based on changes in user emotions. Conventional systems cannot interpret emotions from voice, making it difficult to accurately assess the likelihood of fraud.

[0715] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0716] In this invention, the server includes an audio acquisition means for acquiring the user's call audio, a conversion means for converting the audio data into text data, an analysis means for checking keywords and phrases related to fraud, an emotion analysis means for analyzing emotional information contained in the user's voice, an integrated evaluation means for integrating the analysis results to determine the likelihood of fraud with high accuracy, a recording means for recording the relevant call, and a notification means for issuing a notification. This enables early detection of fraud risk and prompt warning to the user.

[0717] "Voice acquisition means" refers to a device or function for collecting a user's call audio in real time.

[0718] "Conversion means" refers to the process or technology used to convert acquired audio data into text data.

[0719] "Analysis means" refers to a function that analyzes text data and identifies keywords and phrases related to fraud.

[0720] "Emotional analysis methods" refer to technologies for extracting and analyzing emotional information from user voice data.

[0721] The "integrated evaluation method" is a function that integrates analysis results and emotional information to evaluate the likelihood of fraud with high accuracy.

[0722] "Recording means" refers to technology or equipment for recording potentially fraudulent phone calls.

[0723] "Notification methods" refer to features that inform users and their registered family members of potential fraud.

[0724] The system of the present invention has the function of acquiring a user's call audio in real time and highly evaluating the possibility of fraud. When a user starts a call, the terminal captures audio data using an audio acquisition means. This audio data is converted into text data using a conversion means within the terminal. Speech recognition software (e.g., speech recognition API) is used for this conversion.

[0725] The converted text data is sent to the server via a communication network. The server analyzes the text data using a natural language processing model to identify keywords and phrases related to fraud. The server also extracts emotional information from the user's voice using sentiment analysis and uses this information to evaluate the fraud.

[0726] If a call is deemed to be at high risk of fraud, or if the user's emotional state is judged to be outside the normal range, the server will instruct the recording of the call and send a notification to the device containing information on how to access the recording data. Furthermore, this information will be notified to the user and pre-registered family members to enable a swift response.

[0727] For example, a user might receive a call from an unfamiliar caller who emphasizes urgency or speaks in an unstable tone. In this case, the server detects the tone of voice based on sentiment analysis and determines that it is likely to be a scam. The user and their family can then review the recording of the call and take appropriate action if necessary.

[0728] Example prompt for a generative AI model: "Detect changes in the user's emotions during a call and assess the risk of fraud."

[0729] This invention allows users to communicate with greater peace of mind and reduces the risk of becoming a victim of fraud.

[0730] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0731] Step 1:

[0732] When a user initiates a call, the device captures audio data in real time using an audio acquisition device. At this time, the user's call audio is the input, and audio data is generated as the output. Specifically, the device uses a microphone to acquire the audio waveform as digital data.

[0733] Step 2:

[0734] The terminal converts the acquired audio data into text data using a conversion mechanism. Here, the input is audio data, and the output is the corresponding text data. This process utilizes speech recognition software, which analyzes the pattern of the audio waveform and represents its content as text.

[0735] Step 3:

[0736] The terminal sends the converted text data to the server via the communication network. The input is text data, and the output exists as data transferred to the server. In this process, data is encrypted for security purposes to prevent unauthorized access.

[0737] Step 4:

[0738] The server processes the received text data using parsing tools to identify keywords and phrases related to fraud. The input for the parsing is the text data that reaches the server, and the output is an evaluation result indicating the likelihood of fraud. Specifically, a natural language processing model is used to detect specific words and syntax in the data.

[0739] Step 5:

[0740] The server uses emotion analysis tools in parallel with the analysis to extract emotional information from the user's voice. The input to this analysis is variables obtained from the voice data (e.g., tone, pitch), and the output is an evaluation result indicating the user's emotions. Emotion recognition software is used to analyze the voice parameters and identify changes in emotion.

[0741] Step 6:

[0742] The server integrates the analysis results and sentiment analysis results using an integrated evaluation system to determine the likelihood of fraud with high accuracy. The input to this integration is two evaluation results: analysis and sentiment, and the output is an overall fraud risk assessment value. This enables more reliable fraud detection.

[0743] Step 7:

[0744] If a call is deemed highly likely to be fraudulent, the server instructs the device to record the call and sends a notification containing information on how to access the recording data. The input is a fraud risk assessment value, and the output is notification information for the user and pre-registered family members. The server generates an access link to the recording data and includes it in the notification message to ensure rapid information dissemination.

[0745] (Application Example 2)

[0746] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0747] In recent years, with advancements in communication technology, fraudulent activities exploiting voice calls have increased. Therefore, there is a need for a system that allows users to quickly detect potential fraud during a call and take countermeasures. However, conventional fraud detection systems rely solely on keyword and phrase detection, which can result in insufficient accuracy. Furthermore, simply notifying users of potential fraud can worsen their emotional state. Therefore, a system is needed that analyzes changes in the user's emotional state and utilizes this information for fraud detection, enabling more accurate notifications.

[0748] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0749] In this invention, the server includes means for acquiring sound, means for converting it into text information, and means for analyzing emotions to determine the emotional state. This enables a more accurate assessment of the possibility of fraud, taking into account not only the possibility of fraud during a call but also the user's emotional fluctuations. This system also provides support for users to make calls with peace of mind by quickly notifying them of any abnormalities in their emotional state.

[0750] The "acoustic acquisition means" is a device that captures voice information from a user's call in real time and provides it as data necessary for subsequent processing.

[0751] A "conversion means" is a device that has the function of converting acquired audio information into text information, making it available as basic data for analysis.

[0752] An "analysis tool" is a device that analyzes textual information to assess the likelihood of fraud and makes a judgment based on specific conditions.

[0753] An "emotional analysis tool" is a device that has the function of understanding the emotional state of a user during a call and analyzing its changes to evaluate the user's mental state.

[0754] A "recording device" is a device that has the function of recording phone calls in response to the detection of potential fraud or abnormal emotional states, and saving them in a format that can be reviewed at a later date.

[0755] A "notification method" is a function that quickly informs users and registered parties of potential fraud or abnormal emotional states that have been detected.

[0756] The system implementing this invention primarily processes voice data during a call in real time and has the function of evaluating the possibility of fraud and the user's emotional state. The server converts the voice data sent from the user's terminal into text information using the Google Speech-to-Text API. This makes it possible to transform the voice information into a format that is easy to analyze.

[0757] The converted text information is sent to the server, where it is analyzed using a natural language processing model (e.g., BERT or GPT). The purpose of the analysis is to identify vocabulary and expressions related to fraud and to assess the likelihood of fraud. In addition, a sentiment analysis engine such as IBM Watson Tone Analyzer can be used to simultaneously determine the user's emotional state, and if an abnormal change in emotion is detected, the fraud likelihood assessment can be corrected.

[0758] If the server detects potential fraud or emotional instability, it will send alerts to the user and registered stakeholders using notification methods such as Firebase Cloud Messaging. This allows users to quickly recognize the situation and take appropriate action.

[0759] For example, if a user is discussing a real estate transaction over the phone and the other party speaks in an unnatural tone that seems to be pressuring them to make an immediate decision, the system will detect this and analyze the emotional changes that indicate the user's anxiety is increasing. Based on this information, a potential fraud alert will be sent immediately.

[0760] An example of a prompt message is: "Assess the likelihood of fraud based on the user's call content. Also, analyze the user's emotional changes during the call and provide details if anything seems unnatural."

[0761] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0762] Step 1:

[0763] The terminal captures the user's call audio using an acoustic acquisition device. The input is raw audio data, and the output is real-time captured audio data. Specifically, the terminal's microphone collects the audio and converts it into a digital signal.

[0764] Step 2:

[0765] The device converts captured audio data into text information using a conversion mechanism. The input is audio data, and the output is text data. Specifically, the Google Speech-to-Text API is used to analyze the audio data and generate the corresponding text.

[0766] Step 3:

[0767] The server uses analysis tools to evaluate the likelihood of fraud in text data sent from a terminal. The input is converted character information, and the output is the evaluation result regarding the likelihood of fraud. Specifically, a natural language processing model (such as BERT or GPT) analyzes the text and detects vocabulary and expressions related to fraud.

[0768] Step 4:

[0769] The server analyzes the text information and uses sentiment analysis tools to determine the user's emotional state. The input is text data, and the output is the evaluation result of the emotional state. Specifically, IBM Watson Tone Analyzer detects emotional fluctuations from the text and analyzes the results.

[0770] Step 5:

[0771] The server integrates the fraud likelihood assessment and the emotional state assessment to make a final decision. The input is the output results from steps 3 and 4, and the output is the final fraud likelihood assessment and the necessary actions. Specifically, a fraud warning alert is generated based on these two evaluation results.

[0772] Step 6:

[0773] The server uses notification methods to send alerts to users and registered stakeholders. The input is the final fraud warning alert, and the output is the notification sent to users and stakeholders. Specifically, Firebase Cloud Messaging is used to share information by displaying the alert on stakeholders' devices immediately.

[0774] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0775] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0776] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0777] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0778] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0779] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0780] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0781] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0782] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0783] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0784] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0785] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0786] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0787] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0788] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0789] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0790] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0791] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0792] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0793] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0794] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0795] The following is further disclosed regarding the embodiments described above.

[0796] (Claim 1)

[0797] A means for acquiring the voice of a user during a call,

[0798] A conversion means for converting audio data acquired by the aforementioned audio acquisition means into text data,

[0799] An analysis means for analyzing the text data obtained by the conversion means and evaluating the possibility of fraud,

[0800] A recording device for recording calls in the event that a potential fraud is detected,

[0801] A notification system to inform users and registered family members of potential fraud,

[0802] A system that includes this.

[0803] (Claim 2)

[0804] The system according to claim 1, wherein the analysis means executes a natural language processing model that checks for keywords and phrases related to fraud.

[0805] (Claim 3)

[0806] The system according to claim 1, wherein the notification means sends a notification that includes access to a recording file of a call that has been determined to be potentially fraudulent.

[0807] "Example 1"

[0808] (Claim 1)

[0809] A means for acquiring acoustic data to capture user conversations,

[0810] Means for converting the acoustic data acquired by the acoustic data acquisition means into character data,

[0811] A means for analyzing the converted character data and evaluating the possibility of fraudulent activity,

[0812] A means of recording the relevant conversation if it is determined that there is a possibility of misconduct,

[0813] A means of notifying users and registered relatives of potential fraudulent activity,

[0814] A system that includes this.

[0815] (Claim 2)

[0816] The system according to claim 1, wherein the analysis means performs natural language processing techniques to identify words and expressions related to fraudulent activity.

[0817] (Claim 3)

[0818] The system according to claim 1, wherein the notification means transmits a warning that includes access to call recording data of a call that has been determined to be potentially fraudulent.

[0819] "Application Example 1"

[0820] (Claim 1)

[0821] A voice acquisition means for acquiring the user's voice,

[0822] A conversion means for converting audio information acquired by the aforementioned audio acquisition means into text information,

[0823] An analysis means for analyzing the character information obtained by the conversion means and evaluating the possibility of fraud,

[0824] A recording means for recording communications in which fraud is detected,

[0825] A notification system that informs users and registered family members of the possibility of fraud,

[0826] A warning system that detects potential fraud in real time and sends out warnings,

[0827] A system that includes this.

[0828] (Claim 2)

[0829] The system according to claim 1, wherein the analysis means executes a natural language processing model that checks for words and phrases related to fraud.

[0830] (Claim 3)

[0831] The system according to claim 1, wherein the notification means sends a notification that includes access to a record file of communications that have been determined to be potentially fraudulent.

[0832] "Example 2 of combining an emotion engine"

[0833] (Claim 1)

[0834] A means for acquiring the voice of a user during a call,

[0835] A conversion means for converting audio data acquired by the aforementioned audio acquisition means into text data,

[0836] An analysis means for analyzing the text data obtained by the conversion means and evaluating the possibility of fraud,

[0837] A sentiment analysis method that analyzes emotional information contained in the user's voice,

[0838] An integrated evaluation means that evaluates the possibility of fraud using the emotional information obtained by the aforementioned emotional analysis means as a corrective factor,

[0839] A recording means for recording the relevant call when the possibility of fraud is detected,

[0840] A notification system to inform users and registered family members of potential fraud,

[0841] A system that includes this.

[0842] (Claim 2)

[0843] The system according to claim 1, wherein the analysis means executes a natural language processing model that checks for keywords and phrases related to fraud.

[0844] (Claim 3)

[0845] The system according to claim 1, wherein the notification means transmits a notification that includes access to recording data of a call that has been determined to be potentially fraudulent.

[0846] "Application example 2 when combining with an emotional engine"

[0847] (Claim 1)

[0848] A means for acquiring audio from a user's call,

[0849] A conversion means for converting acoustic information acquired by the aforementioned acoustic acquisition means into textual information,

[0850] An analysis means for analyzing the character information obtained by the conversion means and evaluating the possibility of fraud,

[0851] The analytical means is linked to an emotion analysis means that identifies emotional states, and the emotional changes are used for fraud assessment.

[0852] A recording device for recording calls in cases where potential fraud or abnormal emotional states are detected,

[0853] A notification system that alerts users and registered parties to potential fraud or abnormal emotional states,

[0854] A system that includes this.

[0855] (Claim 2)

[0856] The system according to claim 1, wherein the analysis means uses a natural language processing model that checks for vocabulary and expressions related to fraud.

[0857] (Claim 3)

[0858] The system according to claim 1, wherein the notification means transmits a notification that includes access to recording information of a call in which the call is determined to be potentially fraudulent or to be in an abnormal emotional state. [Explanation of symbols]

[0859] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for acquiring the voice of a user during a call, A conversion means for converting audio data acquired by the aforementioned audio acquisition means into text data, An analysis means for analyzing the text data obtained by the conversion means and evaluating the possibility of fraud, A recording device for recording calls in the event that a potential fraud is detected, A notification system to inform users and registered family members of potential fraud, A system that includes this.

2. The system according to claim 1, wherein the analysis means executes a natural language processing model that checks for keywords and phrases related to fraud.

3. The system according to claim 1, wherein the notification means transmits a notification that includes access to a recording file of a call that has been determined to be potentially fraudulent.

Images