system

Abstract

We provide the system. [Solution] A terminal device for collecting audio data, comprising means for recording ambient sounds and converting them into digital data, A means for transmitting the aforementioned digital data using secure communication, A server device that receives the transmitted audio data and stores it in a database, A means for analyzing the aforementioned accumulated audio data to determine the possibility of fraud, A method to automatically send out alerts in case of potential fraud, A system that includes this.

Description

Technical Field

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a persona chatbot control method performed by at least one processor, including steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance as a response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Special fraud targeting the elderly is increasing, and there is a problem that cases where the elderly suffer fraud are endless. Such fraud is carried out by various means, and simple preventive measures and warnings often do not achieve sufficient effects. Therefore, in order to prevent fraud victims, real-time monitoring and immediate response using more advanced technologies are demanded.

Means for Solving the Problems

[0005] This invention provides a means for recording ambient sounds using a terminal device that collects audio data and converting them into digital data. It also includes means for transmitting the converted digital data to a server using secure communication. The server receives the transmitted audio data, stores it in a database, and has means for determining the possibility of fraud through analysis. Furthermore, it includes means for automatically issuing an alert if there is a possibility of fraud. This makes it possible to prevent fraud by detecting and responding to fraudulent activity in real time.

[0006] "Audio data" refers to data that represents collected audio in digital format.

[0007] A "terminal device" is a hardware device used to collect audio and convert it into digital data as needed.

[0008] "Digital data" refers to data obtained by converting audio information into a format that can be processed by a computer.

[0009] "Secure communication" refers to communication that uses protocols and technologies to protect the confidentiality and integrity of data being transmitted.

[0010] A "server device" is a computer system used to receive, analyze, and store audio data.

[0011] A "database" is a system designed to efficiently manage, search, and store large amounts of data.

[0012] "Analysis" is the process of using collected data to detect specific patterns or anomalies.

[0013] "Potential fraud" refers to situations where voice analysis has determined there is a high risk of fraudulent activity occurring.

[0014] An "alert" is a notification that warns a person or system when certain conditions are met. [Brief explanation of the drawing]

[0015] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] This is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] This is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] This is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] This is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] This is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] This is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] This shows an emotion map where multiple emotions are mapped. [Figure 10] This shows an emotion map where multiple emotions are mapped. [Figure 11] This is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] This is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] This is a sequence diagram showing the processing flow of the data processing system in Example 2, which incorporates an emotion engine. [Figure 14] This is a sequence diagram showing the processing flow of the data processing system in Application Example 2, which combines an emotion engine.

Embodiments for Carrying Out the Invention

[0016] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0019] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0020] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, and the like.

[0021] In the following embodiments, the signed communication interface (I / F) is an interface that includes a communication processor and an antenna, etc. The communication interface manages communication between multiple computers. Examples of communication standards applicable to the communication interface include wireless communication standards such as 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark).

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0023] [First Embodiment]

[0024] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0025] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0028] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0031] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0035] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0036] To implement this invention, the user first installs a terminal device. The terminal device is placed within the user's living space and configured to continuously monitor everyday conversations at that location. Voice data is collected in real time by the terminal and converted into a digital format. This converted digital data is then transmitted to a server via secure communication.

[0037] The server stores the received audio data in a database and performs audio analysis based on this data. The audio analysis system within the server converts the audio data into text using natural language processing technology and analyzes whether there is a possibility of fraud based on this text. This analysis involves comparing the audio against a database that stores past fraud patterns.

[0038] As a concrete example, suppose a user is having a phone conversation and keywords such as "cash card" and "PIN" are detected consecutively. The server determines that these keywords match a typical pattern of fraud and immediately identifies it as potentially fraudulent. Based on this result, an alert is automatically sent to designated family members or the police, and a notification is also sent to the user's app. This allows the user and their family to take swift action.

[0039] This system can issue real-time alerts for potential fraudulent situations, preventing victims from becoming victims. Since it requires no special operation from the user, it provides simple yet effective fraud prevention.

[0040] The following describes the processing flow.

[0041] Step 1:

[0042] The device constantly monitors the surrounding audio and records it in real time. The recorded audio is immediately converted into digital data.

[0043] Step 2:

[0044] The terminal packages the converted digital data into packets in a batch format at regular intervals and sends them to the server using a secure communication protocol. Encryption technology is used in this process to prevent data leakage and tampering.

[0045] Step 3:

[0046] The server receives the audio data transmitted from the terminal and first stores it in a database. At this stage, the data is tagged with identification tags so that it can be easily retrieved in subsequent analysis.

[0047] Step 4:

[0048] The AI ​​within the server converts the received audio data into text using natural language processing technology. The text data is then passed to an analysis module, which compares it against a database of past fraud patterns.

[0049] Step 5:

[0050] Based on the analysis results, the server activates the alert generation module if it determines that there is a high probability of fraud. The alert information is immediately prepared and passed on to the subsequent alert generation step.

[0051] Step 6:

[0052] The server automatically sends an alert to the relevant family members or police contacts. This alert includes information about the content and date of the suspected fraudulent conversation, as well as specific details about the likelihood of fraud.

[0053] Step 7:

[0054] Users will be notified of potential fraud through apps on their smartphones, etc. Upon receiving a notification, users can check the situation, seek support from family or the police if necessary, and take prompt action against the fraud.

[0055] (Example 1)

[0056] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0057] In modern society, fraudulent activities exploiting communication methods are on the rise, with a particularly high number of incidents targeting the elderly and those vulnerable to information overload. Such fraud poses a serious threat to personal property and safety, making early detection and prevention crucial. However, conventional measures struggle to detect fraudulent activities in real time and respond quickly, resulting in numerous victims. To address this problem, an effective and efficient fraud detection and warning system is necessary.

[0058] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0059] In this invention, the server includes means for converting audio signals into text data using natural language processing technology, means for determining the possibility of fraudulent activity by comparing it with existing information storage, and means for automatically issuing a warning if there is a possibility of fraudulent activity. This makes it possible to analyze the possibility of fraud in real time and issue a warning immediately.

[0060] "Audio signals" refer to information obtained by converting sound vibrations emitted by the environment or people into an electrical format.

[0061] A "communication terminal" is a device used to transmit voice signals to external devices or systems.

[0062] A "digital format" refers to a format in which analog information is represented using a combination of binary numbers that can be processed by a computer.

[0063] "Protective communication technology" refers to communication technology that uses encryption technology and other methods to protect data from interception and tampering by third parties during transmission and reception.

[0064] An "information processing device" refers to a device that has the ability to analyze, store, or transmit received data.

[0065] A "storage device" refers to hardware or digital media used to store information for extended periods.

[0066] "Fraudulent activity" refers to the act of intentionally providing false information with the intent to deceive others and thereby illegally obtaining some kind of benefit.

[0067] "Natural language processing technology" refers to the techniques and algorithms that enable computers to understand and process human language.

[0068] "Text data" refers to the form of strings or sentences obtained by converting audio information into a human-readable format.

[0069] "Information storage" refers to devices or services for storing and managing data.

[0070] A "warning" refers to a notification or alert intended to inform someone of some kind of danger or caution.

[0071] To implement this invention, the user first needs to install a communication terminal in their living space, such as their home or office. The terminal is a device capable of continuously collecting ambient audio signals and converting them into a digital format. By using audio signal processing hardware installed in the terminal, such as a DSP (Digital Signal Processor), analog audio is converted into digital data.

[0072] The converted digital data is transmitted from the terminal to the server using secure communication technology, such as SSL / TLS encryption. The server stores the received data in its information processing unit and prepares it for subsequent analysis. This analysis uses methods to convert speech into text data using natural language processing technology. Specifically, speech recognition software and APIs (Application Programming Interfaces) are used.

[0073] The server compares the obtained text data with existing fraud pattern information. Through this comparison, it is possible to identify potentially fraudulent conversation content. For detected fraudulent activity, the server automatically issues a warning and sends a notification to the user's smartphone app or designated contacts. This process allows users to prevent potential fraud.

[0074] As a concrete example, consider a scenario where a user is receiving a suspicious phone call and their device identifies keywords such as "bank" and "PIN." In such a situation, the server would determine that these keyword patterns are potentially fraudulent and immediately issue a warning. This would allow the user and their family to take swift action based on the notification.

[0075] An example of a prompt statement that utilizes a generative AI model is the question, "Please explain in detail the process from collecting voice data in a living space to detecting fraud." By using this prompt statement, the AI ​​can generate a detailed process explanation, providing useful information for system operation.

[0076] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0077] Step 1:

[0078] The user installs a communication terminal in their living space. The terminal uses a high-sensitivity microphone to collect ambient audio signals in real time. Inputs include ambient sounds and human conversations. The terminal uses an audio signal processing device to convert these audio signals into a digital format and generate digital data as output.

[0079] Step 2:

[0080] The terminal transmits the generated digital data to the server using secure communication technology. Specifically, the digital data is encrypted using the SSL / TLS protocol. The input is digital audio data, and the output is encrypted data that the server can receive.

[0081] Step 3:

[0082] The server decodes the digital data received from the terminal and stores it in the information processing device. It receives encrypted data as input, decodes it, and then produces audio data that can be stored in the database as output.

[0083] Step 4:

[0084] The server converts stored audio data into text using natural language processing technology. It takes audio data as input, generates character data via a speech recognition engine, and outputs accurately transcribed text.

[0085] Step 5:

[0086] The server analyzes the converted text data by comparing it to existing fraud patterns. It takes text data as input and uses topic modeling and keyword matching algorithms to determine the likelihood of fraud. If fraud is highly likely, it provides a flag indicating this.

[0087] Step 6:

[0088] The server issues a warning if it determines that something is potentially fraudulent. It receives a fraud warning flag as input and sends a message to the user's smartphone app or registered contacts via its notification system. As output, it generates an automatically sent warning notification.

[0089] (Application Example 1)

[0090] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0091] In modern society, fraud is becoming increasingly sophisticated, especially fraudulent activities conducted through verbal communication. To prevent such fraud, an effective system is needed that monitors audio in real time and immediately detects potential fraud. Furthermore, if potential fraud is detected, a mechanism is needed to quickly notify those involved and minimize the damage.

[0092] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0093] In this invention, the server includes means for acquiring voice information and converting it into digital information, means for transmitting the digital information using information-protected communication, and means for analyzing the voice information to determine the possibility of fraud and automatically sending a notification. This makes it possible to detect the possibility of fraud with high accuracy and speed and notify relevant parties in real time.

[0094] "Audio information" refers to data obtained by converting ambient sounds into a digital format.

[0095] An "information processing device" is a device that acquires sound and processes it as digital data.

[0096] "Digital information" refers to data in which audio is represented in a digital format.

[0097] "Information protection communication" refers to a communication method for securely transmitting digital data.

[0098] A "processing unit" is a device that analyzes received audio information and stores it on a recording medium.

[0099] A "recording medium" is a physical or electronic medium used to store data.

[0100] "Natural language processing" is a technique for converting audio data into text and analyzing it.

[0101] "Potential fraud" refers to signs that the observed audio data may be related to fraudulent activity.

[0102] "Notification" refers to the act of sending a warning to a user or relevant party when certain conditions are met.

[0103] A "communication device" is a device that analyzes voice data in real time and transmits the results.

[0104] In implementing this invention, the following system configuration is used. A terminal device is placed in the user's surrounding environment and acquires voice information through a microphone. This voice is converted into digital data and transmitted to a server via secure communication. In this process, the Google® Cloud Speech-to-Text API is used to convert the voice to text. The server uses natural language processing to analyze the received text data and determine whether it contains fraudulent keywords. At this stage, the Hugging Face Transformers library is used.

[0105] If a potential scam is detected, the server uses the Twilio API to send a notification to the user, their family, and related parties. This notification allows the user to be aware of the potential scam in real time and take immediate action.

[0106] For example, if a user receives a phone call and keywords such as "PIN" or "bank account number" are uttered during the conversation, the system will immediately identify this as a sign of fraud and send a notification. This process is performed automatically and does not require any special interface interaction from the user.

[0107] The following sentences may be used as prompts for generative AI models:

[0108] Please provide a list of phrases commonly used in scams.

[0109] "How can you detect signs of fraud from everyday conversations?"

[0110] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0111] Step 1:

[0112] The device uses a microphone to acquire ambient audio information. The input audio data is an analog signal, which is then converted into digital data. Specifically, the audio waveform is digitized by sampling via audio recording software.

[0113] Step 2:

[0114] The terminal transmits digital data to the server using secure communication. The input is the digital voice data obtained in the previous step, which is encrypted before transmission. It is common practice to use the standard SSL / TLS protocol to ensure data security.

[0115] Step 3:

[0116] The server converts received digital audio data into text data using the Google Cloud Speech-to-Text API. The input is encrypted digital audio data, speech recognition is performed via the API, and text data is generated as output. Specifically, this involves analyzing the audio waveform and phoneme recognition.

[0117] Step 4:

[0118] The server analyzes the generated text data using natural language processing methods. In this step, the Hugging Face Transformers library is used to evaluate whether the input text data contains fraud-related keywords. The specific actions performed here are matching the text with word patterns and statistical analysis.

[0119] Step 5:

[0120] The server uses the Twilio API to send notifications when it detects potential fraud. The input is the text data that was determined to be highly fraudulent in the previous step, and the output is a warning notification sent to the user or designated parties. The notification can be sent via SMS or email.

[0121] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0122] This invention provides a system that incorporates an emotion engine to recognize emotions from user voice data, thereby achieving advanced fraud prevention. First, the terminal records the sounds around the user and converts them into digital data. The converted voice data is transmitted to the server via secure communication. The server stores the voice data in a database and extracts emotional information using the emotion engine along with basic voice analysis.

[0123] This emotion engine analyzes the tone, speed, and volume of the voice to identify the user's emotional state (e.g., anxiety, anger, relief). Based on this, a system is built to accurately determine the likelihood of fraud. For example, if a user emphasizes keywords such as "urgent" or "important" and also indicates anxiety, the server evaluates this as a high risk of fraud and increases the urgency of the alert.

[0124] For example, if a user is asked over the phone to "provide details of their cash card immediately" and is feeling anxious, the emotion engine will detect this anxiety. The server will then take this anxiety into consideration and send a particularly urgent alert to family members or the police. The notification will include information that the user is showing a stronger-than-usual emotional reaction, prompting a quick response.

[0125] In this way, a system that incorporates an emotion engine allows users to receive feedback on their own emotional state while simultaneously strengthening their crime prevention measures. This system can provide advanced support for preventing fraud and further ensure user safety.

[0126] The following describes the processing flow.

[0127] Step 1:

[0128] The device monitors the surrounding audio in real time and starts recording. The recorded audio is immediately converted into digital data. This digital data is then prepared to be sent to the server via secure communication.

[0129] Step 2:

[0130] The server stores the received audio data in a database. The data is saved in an identifiable format and used for subsequent analysis.

[0131] Step 3:

[0132] An analysis module on the server converts the audio data into text using natural language processing technology, and based on this information, it begins detecting basic fraud patterns. Simultaneously, an emotion engine analyzes the audio data and identifies the user's emotional state from information such as tone and speed of speech.

[0133] Step 4:

[0134] The server detects suspicious interactions by comparing them with a database of fraud patterns and transcribed data. It also considers the results of sentiment analysis and reassessss the risk of fraud if the user is showing anxiety or other strong emotions. This process calculates a fraud likelihood score and determines the urgency of the situation.

[0135] Step 5:

[0136] If a fraudulent activity is deemed highly likely, the server generates an alert, taking into account the results of sentiment analysis. The content and priority of the alert are set according to the urgency and the user's emotional state.

[0137] Step 6:

[0138] The server sends the generated alerts to pre-designated family members or police contacts. The notifications include information about the potential for fraud detected, the user's emotional response, and emergency actions to take.

[0139] Step 7:

[0140] Users receive alert notifications through a smartphone app. These notifications include specific actions to take and, if necessary, contact information for support, enabling quick action.

[0141] (Example 2)

[0142] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0143] In modern society, voice-based fraud is on the rise, and effective preventative measures are needed. However, conventional systems have struggled to analyze emotional information with sufficient accuracy to detect fraudulent activity. Furthermore, they are unable to provide real-time responses based on the user's emotional state, making it impossible to issue effective warnings before fraud is committed.

[0144] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0145] In this invention, the server includes an information processing device means for receiving voice data and storing it in a database, a means for analyzing the intonation, speed, and intensity of the voice and extracting emotional information, and a means for determining the possibility of fraud based on the emotional information and issuing a warning. This enables high-precision evaluation of the risk of fraud experienced by the user through voice and prompt issuance of warnings.

[0146] "Audio data" refers to information used to record and process audio in a digital format.

[0147] A "terminal device" is a device that has the function of recording audio and converting it into digital data.

[0148] "Secure communication" is a technology that provides encrypted and safe communication paths to prevent third parties from eavesdropping on or tampering with data.

[0149] An "information processing device" is a device used to process digital data using a computer.

[0150] A "database" is a system that efficiently stores and manages large amounts of digital information in a searchable format.

[0151] "Voice intonation, speed, and volume" refers to the pitch, speed, and volume of speech, and are elements used to identify emotions.

[0152] "Emotional information" refers to data that indicates a person's emotional state, analyzed from audio data.

[0153] "Potential fraud" refers to the likelihood of fraudulent activity occurring, as determined based on audio data and emotional information.

[0154] A "warning" is an alert sent to users or related parties when a situation is deemed highly likely to be fraudulent.

[0155] This invention is a system that extracts emotional information from a user's voice and determines the likelihood of fraud with high accuracy. This system mainly consists of a terminal device and a server.

[0156] The device records the sounds around the user and converts the recorded audio into digital data. This is done using a built-in microphone and audio recording software. The converted digital data is sent to the server via a secure communication protocol. TLS is a suitable communication protocol for this purpose.

[0157] The server stores the received audio data in a database and performs basic analysis using audio analysis software. Subsequently, the server uses an emotion engine to analyze the intonation, speed, and volume of the speech, extracting the user's emotional information. This emotional information is then analyzed using a machine learning model, which is based on TENSORFLOW® and PyTorch.

[0158] When assessing fraud risk, the server analyzes extracted sentiment information and combines specific keywords with emotional states to determine the likelihood of fraud. Based on this assessment, a warning is generated according to the urgency of the situation and notified family members and relevant parties. Notification methods include email, SMS, and push notifications via a dedicated app.

[0159] For example, if a user calls and says, "I want to know the details of my cash card," and an anxious feeling is detected, the emotion engine will detect that anxiety. The server will then determine this to be high risk and immediately issue a warning.

[0160] An example of a prompt for a generative AI model is, "Explain how many steps are taken to process the audio data and how the alert is ultimately generated."

[0161] This system allows users to receive advance warnings about potential fraud risks, thereby enhancing their personal safety.

[0162] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0163] Step 1:

[0164] The device records ambient sound around the user and converts it into digital data. The input is ambient sound, and the output is digital audio data. Specifically, it collects sound using a built-in microphone and captures the data with audio recording software. An ADC (analog-to-digital converter) is used to convert the recorded analog audio into a digital signal.

[0165] Step 2:

[0166] The terminal sends this digital audio data to the server using a secure communication protocol (e.g., TLS). The input is the digital audio data, and the output is the completion of secure data transmission to the server. Specifically, the terminal encrypts the data and transfers it to the server via the transmission protocol.

[0167] Step 3:

[0168] The server receives transmitted audio data and stores it in a database. The input is digital audio data from the terminal, and the output is the stored database entry. The server processes the received data and stores it in a high-speed database management system.

[0169] Step 4:

[0170] The server runs acoustic analysis software to analyze the stored audio data and extract the basic characteristics of the sound. The input is the stored digital audio data, and the output is the result of the audio analysis. Specifically, the server calculates the spectral characteristics and basic sound patterns of the audio.

[0171] Step 5:

[0172] The server uses an emotion engine to identify emotional information from the tone, speed, and volume of speech. The input is the result of speech analysis, and the output is emotional information. Specifically, it analyzes emotions using machine learning models (e.g., TensorFlow or PyTorch).

[0173] Step 6:

[0174] The server assesses the likelihood of fraud based on extracted sentiment information and issues warnings as necessary. Inputs are sentiment information and keywords from the audio, while output is a warning notification. If the server determines a high risk, it generates an alert and notifies relevant parties via email, SMS, etc.

[0175] (Application Example 2)

[0176] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0177] The problem that this invention aims to solve is to prevent fraud and illegal activities that may occur when using electronic payments by analyzing the emotional state of the user, thereby ensuring security so that transactions can be conducted with peace of mind.

[0178] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0179] In this invention, the server includes means for receiving voice information and storing it in a data storage device, means for analyzing the stored voice information to determine the risk of a transaction, and means for identifying the user's emotional state and confirming the security of the transaction based on that information. This makes it possible to assess the risk of fraud in real time and issue appropriate alerts to the user.

[0180] A "device for collecting audio data" is a terminal that has the function of recording sounds around the user and converting them into digital information.

[0181] "Secure communication" refers to a communication method that uses security protocols to transmit digital information without it being intercepted by a third party.

[0182] An "information processing device" is a computer system that stores received audio information in a database and analyzes that information as needed.

[0183] "Means for determining the risk of a transaction" refers to algorithms and processes that analyze accumulated voice information to evaluate the possibility of fraud or illegal activity.

[0184] A "means for identifying a user's emotional state" refers to a system that has the function of identifying a user's emotions by analyzing tone, speed, intensity, etc., from audio data.

[0185] "Methods for automatically issuing warnings" refers to a system that has the function of quickly sending notifications to users and related parties when a transaction is deemed to be risky.

[0186] The system for carrying out the present invention consists of a device for collecting voice data, a communication means for securely transmitting information, and an information processing device for storing and analyzing voice information. This system is realized by collecting ambient sounds and converting them into digital voice data when a user makes an electronic payment.

[0187] The device that collects voice data operates using an application installed on mobile devices such as smartphones and tablets. It records the user's speech, converts it into digital voice data, and then transmits the data to a server using secure protocols such as SSL and TLS. This application also analyzes emotions from the voice using a generative AI model.

[0188] The server stores the received audio data in a data storage device and simultaneously processes the data using audio analysis technology. Specifically, an AI-powered emotion analysis engine analyzes the tone, speed, and volume of the voice to determine the user's emotional state. Based on this information, the system decides how to act during the electronic payment process. For example, if a user shows anxiety during a transaction, the server immediately evaluates the situation and sends a notification warning of the possibility of fraud.

[0189] This system is used to verify the safety of purchases made on e-commerce sites that users normally use. For example, by providing voice information such as, "Is it really safe to proceed with the purchase of this product?", the system analyzes the information and determines whether it is safe.

[0190] An example of a prompt message for a generative AI model is: "Analyze the voice data and create an alert indicating how to perform security checks on electronic payment transactions if the user's emotional state is anxious." This prompt is used to support real-time feedback functionality based on sentiment analysis.

[0191] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0192] Step 1:

[0193] The device collects ambient sound from the user's surroundings and converts it into digital audio data. The input is ambient sound, which is recorded using a microphone and converted into a digital format output. The converted digital audio data is then prepared for transmission to a server via secure communication.

[0194] Step 2:

[0195] The terminal sends the converted digital audio data to the server using a secure protocol (e.g., SSL / TLS). The input is the digital audio data generated in step 1, and the output arrives securely at the server as encrypted data. This ensures data security and privacy.

[0196] Step 3:

[0197] The server stores the received digital audio data in a database. The input is the encrypted data sent in step 2, the secure communication is decrypted and the digital audio data is converted into an internal format, and the output becomes digital information that is securely stored in the database.

[0198] Step 4:

[0199] The server analyzes the stored audio data. This process includes a generative AI model that uses AI to analyze the tone, speed, and volume of the voice to determine the user's emotional state. The input is the audio data stored in the database in step 3, and the output is information representing the emotional state.

[0200] Step 5:

[0201] The server verifies the security of the transaction based on the analysis results. The input is the sentiment state information obtained in step 4, and by performing a risk assessment according to the sentiment, the server generates an output indicating whether the security of the transaction has been ensured.

[0202] Step 6:

[0203] If the server detects any signs of unease, it will send a warning to the user. The input is the risk assessment result from step 5, and the output is a warning message displayed on the user's terminal. This prompts the user to re-evaluate whether to continue trading.

[0204] Step 7:

[0205] The user reviews the warning displayed on the terminal and reconfirms the security of electronic payments. The input is the warning message received in step 6, and the output prompts the user to take action based on the information needed to determine the specific actions and responses they should take.

[0206] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0207] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0208] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0209] [Second Embodiment]

[0210] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0211] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0212] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0213] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0214] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0215] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0216] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0217] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0218] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0219] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0220] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0221] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0222] To implement this invention, the user first installs a terminal device. The terminal device is placed within the user's living space and configured to continuously monitor everyday conversations at that location. Voice data is collected in real time by the terminal and converted into a digital format. This converted digital data is then transmitted to a server via secure communication.

[0223] The server stores the received audio data in a database and performs audio analysis based on this data. The audio analysis system within the server converts the audio data into text using natural language processing technology and analyzes whether there is a possibility of fraud based on this text. This analysis involves comparing the audio against a database that stores past fraud patterns.

[0224] As a concrete example, suppose a user is having a phone conversation and keywords such as "cash card" and "PIN" are detected consecutively. The server determines that these keywords match a typical pattern of fraud and immediately identifies it as potentially fraudulent. Based on this result, an alert is automatically sent to designated family members or the police, and a notification is also sent to the user's app. This allows the user and their family to take swift action.

[0225] This system can issue real-time alerts for potential fraudulent situations, preventing victims from becoming victims. Since it requires no special operation from the user, it provides simple yet effective fraud prevention.

[0226] The following describes the processing flow.

[0227] Step 1:

[0228] The device constantly monitors the surrounding audio and records it in real time. The recorded audio is immediately converted into digital data.

[0229] Step 2:

[0230] The terminal packages the converted digital data into packets in a batch format at regular intervals and sends them to the server using a secure communication protocol. Encryption technology is used in this process to prevent data leakage and tampering.

[0231] Step 3:

[0232] The server receives the audio data transmitted from the terminal and first stores it in a database. At this stage, the data is tagged with identification tags so that it can be easily retrieved in subsequent analysis.

[0233] Step 4:

[0234] The AI ​​within the server converts the received audio data into text using natural language processing technology. The text data is then passed to an analysis module, which compares it against a database of past fraud patterns.

[0235] Step 5:

[0236] Based on the analysis results, the server activates the alert generation module if it determines that there is a high probability of fraud. The alert information is immediately prepared and passed on to the subsequent alert generation step.

[0237] Step 6:

[0238] The server automatically sends an alert to the relevant family members or police contacts. This alert includes information about the content and date of the suspected fraudulent conversation, as well as specific details about the likelihood of fraud.

[0239] Step 7:

[0240] Users will be notified of potential fraud through apps on their smartphones, etc. Upon receiving a notification, users can check the situation, seek support from family or the police if necessary, and take prompt action against the fraud.

[0241] (Example 1)

[0242] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0243] In modern society, fraudulent activities exploiting communication methods are on the rise, with a particularly high number of incidents targeting the elderly and those vulnerable to information overload. Such fraud poses a serious threat to personal property and safety, making early detection and prevention crucial. However, conventional measures struggle to detect fraudulent activities in real time and respond quickly, resulting in numerous victims. To address this problem, an effective and efficient fraud detection and warning system is necessary.

[0244] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0245] In this invention, the server includes means for converting audio signals into text data using natural language processing technology, means for determining the possibility of fraudulent activity by comparing it with existing information storage, and means for automatically issuing a warning if there is a possibility of fraudulent activity. This makes it possible to analyze the possibility of fraud in real time and issue a warning immediately.

[0246] "Audio signals" refer to information obtained by converting sound vibrations emitted by the environment or people into an electrical format.

[0247] A "communication terminal" is a device used to transmit voice signals to external devices or systems.

[0248] A "digital format" refers to a format in which analog information is represented using a combination of binary numbers that can be processed by a computer.

[0249] "Protective communication technology" refers to communication technology that uses encryption technology and other methods to protect data from interception and tampering by third parties during transmission and reception.

[0250] An "information processing device" refers to a device that has the ability to analyze, store, or transmit received data.

[0251] A "storage device" refers to hardware or digital media used to store information for extended periods.

[0252] "Fraudulent activity" refers to the act of intentionally providing false information with the intent to deceive others and thereby illegally obtaining some kind of benefit.

[0253] "Natural language processing technology" refers to the techniques and algorithms that enable computers to understand and process human language.

[0254] "Text data" refers to the form of strings or sentences obtained by converting audio information into a human-readable format.

[0255] "Information storage" refers to devices or services for storing and managing data.

[0256] A "warning" refers to a notification or alert intended to inform someone of some kind of danger or caution.

[0257] To implement this invention, the user first needs to install a communication terminal in their living space, such as their home or office. The terminal is a device capable of continuously collecting ambient audio signals and converting them into a digital format. By using audio signal processing hardware installed in the terminal, such as a DSP (Digital Signal Processor), analog audio is converted into digital data.

[0258] The converted digital data is transmitted from the terminal to the server using secure communication technology, such as SSL / TLS encryption. The server stores the received data in its information processing unit and prepares it for subsequent analysis. This analysis uses methods to convert speech into text data using natural language processing technology. Specifically, speech recognition software and APIs (Application Programming Interfaces) are used.

[0259] The server compares the obtained text data with existing fraud pattern information. Through this comparison, it is possible to identify potentially fraudulent conversation content. For detected fraudulent activity, the server automatically issues a warning and sends a notification to the user's smartphone app or designated contacts. This process allows users to prevent potential fraud.

[0260] As a concrete example, consider a scenario where a user is receiving a suspicious phone call and their device identifies keywords such as "bank" and "PIN." In such a situation, the server would determine that these keyword patterns are potentially fraudulent and immediately issue a warning. This would allow the user and their family to take swift action based on the notification.

[0261] An example of a prompt statement that utilizes a generative AI model is the question, "Please explain in detail the process from collecting voice data in a living space to detecting fraud." By using this prompt statement, the AI ​​can generate a detailed process explanation, providing useful information for system operation.

[0262] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0263] Step 1:

[0264] The user installs a communication terminal in their living space. The terminal uses a high-sensitivity microphone to collect ambient audio signals in real time. Inputs include ambient sounds and human conversations. The terminal uses an audio signal processing device to convert these audio signals into a digital format and generate digital data as output.

[0265] Step 2:

[0266] The terminal transmits the generated digital data to the server using secure communication technology. Specifically, the digital data is encrypted using the SSL / TLS protocol. The input is digital audio data, and the output is encrypted data that the server can receive.

[0267] Step 3:

[0268] The server decodes the digital data received from the terminal and stores it in the information processing device. It receives encrypted data as input, decodes it, and then produces audio data that can be stored in the database as output.

[0269] Step 4:

[0270] The server converts stored audio data into text using natural language processing technology. It takes audio data as input, generates character data via a speech recognition engine, and outputs accurately transcribed text.

[0271] Step 5:

[0272] The server analyzes the converted text data by comparing it to existing fraud patterns. It takes text data as input and uses topic modeling and keyword matching algorithms to determine the likelihood of fraud. If fraud is highly likely, it provides a flag indicating this.

[0273] Step 6:

[0274] The server issues a warning if it determines that something is potentially fraudulent. It receives a fraud warning flag as input and sends a message to the user's smartphone app or registered contacts via its notification system. As output, it generates an automatically sent warning notification.

[0275] (Application Example 1)

[0276] Next, Application Example 1 will be described. In the following description, the data processing device 12 is referred to as a "server", and the smart glasses 214 are referred to as a "terminal".

[0277] In modern society, fraud crimes have become sophisticated, especially fraud committed through oral communication is sophisticated. In order to prevent the damage of such fraud, an effective system that monitors voices in real time and immediately detects the possibility of fraud is required. In addition, when the possibility of fraud is detected, a mechanism that quickly notifies the relevant parties and minimizes the damage is necessary.

[0278] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0279] In this invention, the server includes a constituent means for acquiring voice information and converting it into digital information, a constituent means for transmitting the digital information using information protection communication, and a constituent means for analyzing the voice information to judge the possibility of fraud and automatically transmitting a notification. As a result, it becomes possible to detect the possibility of fraud with high accuracy and quickly, and notify the relevant parties in real time.

[0280] "Voice information" is data obtained by converting ambient sounds into a digital format.

[0281] [[ID=-- -- --20]]"Information processing device" is a device for acquiring voices and processing them as digital data. -- -- -->

[0282] "Digital information" is data representing voices in a digital format.

[0283] "Information protection communication" is a communication means for securely transmitting digital data.

[0284] "Arithmetic device" is a device for analyzing received voice information and storing it in a recording medium. [[ID=-- -- --34]]

[0285] [[ID=-- -- --35]] It seems there are some line breaks in the original text that might be a bit irregular in terms of the overall structure. I've translated it as accurately as possible while maintaining the line breaks as per the instructions. If you have any further questions or need clarification, feel free to ask."Recording medium" refers to a physical or electronic medium for storing data.

[0286] "Natural language processing method" refers to a technology for converting voice data into text and analyzing it.

[0287] "Possibility of fraud" refers to signs indicating the possibility that the observed voice data is involved in fraud.

[0288] "Notification" refers to the act of sending a warning to users or related parties when specific conditions are met.

[0289] "Communication device" refers to a device for analyzing voice data in real time and transmitting the results.

[0290] In implementing this invention, the following system configuration is used. The terminal device is placed in the user's surrounding environment and acquires voice information through a microphone. This voice is converted into digital data and transmitted to the server via information-protected communication. In this process, the Google Cloud Speech-to-Text API is used to convert the voice into text. The server analyzes the received text data using the natural language processing method to determine whether fraud keywords are included. At this stage, the Transformers library of Hugging Face is used.

[0291] If it is determined that there is a possibility of fraud, the server uses the Twilio API to send a notification to the user and their family or related parties. With this notification, the user can know the possibility of fraud in real time and respond promptly.

[0292] As a specific example, when the user answers a call and keywords such as "password" or "bank account number" are uttered during the conversation, the system immediately identifies this as a sign of fraud and sends a notification. This process is automatically executed, and the user does not need to perform special interface operations.

[0293] The following sentences may be used as prompts for generative AI models:

[0294] Please provide a list of phrases commonly used in scams.

[0295] "How can you detect signs of fraud from everyday conversations?"

[0296] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0297] Step 1:

[0298] The device uses a microphone to acquire ambient audio information. The input audio data is an analog signal, which is then converted into digital data. Specifically, the audio waveform is digitized by sampling via audio recording software.

[0299] Step 2:

[0300] The terminal transmits digital data to the server using secure communication. The input is the digital voice data obtained in the previous step, which is encrypted before transmission. It is common practice to use the standard SSL / TLS protocol to ensure data security.

[0301] Step 3:

[0302] The server converts received digital audio data into text data using the Google Cloud Speech-to-Text API. The input is encrypted digital audio data, speech recognition is performed via the API, and text data is generated as output. Specifically, this involves analyzing the audio waveform and phoneme recognition.

[0303] Step 4:

[0304] The server analyzes the generated text data using natural language processing methods. In this step, it evaluates whether the input text data contains fraud-related keywords using the Hugging Face Transformers library. The specific operations performed here are the matching with word patterns in the text and statistical analysis.

[0305] Step 5:

[0306] If the server recognizes the possibility of fraud, it sends a notification using the Twilio API. The input is the text data determined to have a high possibility of fraud in the previous step, and a warning notification for the user or specified parties is generated as the output. The specific form of the notification is SMS or email.

[0307] Furthermore, an emotion engine for estimating the user's emotion may be combined. That is, the specific processing unit 290 may estimate the user's emotion using the emotion recognition model 59 and perform specific processing using the user's emotion.

[0308] The present invention is a system incorporating an emotion engine that recognizes emotions from the user's voice data and realizes advanced fraud prevention. First, the terminal records the surrounding voice of the user and converts it into digital data. The converted voice data is transmitted to the server via secure communication. The server stores the voice data in a database and extracts emotion information using the emotion engine along with basic voice analysis.

[0309] This emotion engine analyzes the tone, speed, intensity, etc. of the voice and identifies the user's emotional state (e.g., anxiety, anger, relief, etc.). Based on this, an auxiliary means for accurately judging the possibility of fraud is constructed. For example, if the user emphasizes keywords such as "hurry" and "important" and shows an anxious emotion, the server evaluates this as a high risk of fraud and raises the urgency of the alert.

[0310] For example, if a user is asked over the phone to "provide details of their cash card immediately" and is feeling anxious, the emotion engine will detect this anxiety. The server will then take this anxiety into consideration and send a particularly urgent alert to family members or the police. The notification will include information that the user is showing a stronger-than-usual emotional reaction, prompting a quick response.

[0311] In this way, a system that incorporates an emotion engine allows users to receive feedback on their own emotional state while simultaneously strengthening their crime prevention measures. This system can provide advanced support for preventing fraud and further ensure user safety.

[0312] The following describes the processing flow.

[0313] Step 1:

[0314] The device monitors the surrounding audio in real time and starts recording. The recorded audio is immediately converted into digital data. This digital data is then prepared to be sent to the server via secure communication.

[0315] Step 2:

[0316] The server stores the received audio data in a database. The data is saved in an identifiable format and used for subsequent analysis.

[0317] Step 3:

[0318] An analysis module on the server converts the audio data into text using natural language processing technology, and based on this information, it begins detecting basic fraud patterns. Simultaneously, an emotion engine analyzes the audio data and identifies the user's emotional state from information such as tone and speed of speech.

[0319] Step 4:

[0320] The server detects suspicious interactions by comparing them with a database of fraud patterns and transcribed data. It also considers the results of sentiment analysis and reassessss the risk of fraud if the user is showing anxiety or other strong emotions. This process calculates a fraud likelihood score and determines the urgency of the situation.

[0321] Step 5:

[0322] If a fraudulent activity is deemed highly likely, the server generates an alert, taking into account the results of sentiment analysis. The content and priority of the alert are set according to the urgency and the user's emotional state.

[0323] Step 6:

[0324] The server sends the generated alerts to pre-designated family members or police contacts. The notifications include information about the potential for fraud detected, the user's emotional response, and emergency actions to take.

[0325] Step 7:

[0326] Users receive alert notifications through a smartphone app. These notifications include specific actions to take and, if necessary, contact information for support, enabling quick action.

[0327] (Example 2)

[0328] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0329] In modern society, voice-based fraud is on the rise, and effective preventative measures are needed. However, conventional systems have struggled to analyze emotional information with sufficient accuracy to detect fraudulent activity. Furthermore, they are unable to provide real-time responses based on the user's emotional state, making it impossible to issue effective warnings before fraud is committed.

[0330] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0331] In this invention, the server includes an information processing device means for receiving voice data and storing it in a database, a means for analyzing the intonation, speed, and intensity of the voice and extracting emotional information, and a means for determining the possibility of fraud based on the emotional information and issuing a warning. This enables high-precision evaluation of the risk of fraud experienced by the user through voice and prompt issuance of warnings.

[0332] "Audio data" refers to information used to record and process audio in a digital format.

[0333] A "terminal device" is a device that has the function of recording audio and converting it into digital data.

[0334] "Secure communication" is a technology that provides encrypted and safe communication paths to prevent third parties from eavesdropping on or tampering with data.

[0335] An "information processing device" is a device used to process digital data using a computer.

[0336] A "database" is a system that efficiently stores and manages large amounts of digital information in a searchable format.

[0337] "Voice intonation, speed, and volume" refers to the pitch, speed, and volume of speech, and are elements used to identify emotions.

[0338] "Emotional information" refers to data that indicates a person's emotional state, analyzed from audio data.

[0339] "Potential fraud" refers to the likelihood of fraudulent activity occurring, as determined based on audio data and emotional information.

[0340] A "warning" is an alert sent to users or related parties when a situation is deemed highly likely to be fraudulent.

[0341] This invention is a system that extracts emotional information from a user's voice and determines the likelihood of fraud with high accuracy. This system mainly consists of a terminal device and a server.

[0342] The device records the sounds around the user and converts the recorded audio into digital data. This is done using a built-in microphone and audio recording software. The converted digital data is sent to the server via a secure communication protocol. TLS is a suitable communication protocol for this purpose.

[0343] The server stores the received audio data in a database and performs basic analysis using audio analysis software. Subsequently, the server uses an emotion engine to analyze the intonation, speed, and volume of the speech, extracting the user's emotional information. This emotional information is then analyzed using a machine learning model, which is based on TensorFlow or PyTorch.

[0344] When assessing fraud risk, the server analyzes extracted sentiment information and combines specific keywords with emotional states to determine the likelihood of fraud. Based on this assessment, a warning is generated according to the urgency of the situation and notified family members and relevant parties. Notification methods include email, SMS, and push notifications via a dedicated app.

[0345] For example, if a user calls and says, "I want to know the details of my cash card," and an anxious feeling is detected, the emotion engine will detect that anxiety. The server will then determine this to be high risk and immediately issue a warning.

[0346] An example of a prompt for a generative AI model is, "Explain how many steps are taken to process the audio data and how the alert is ultimately generated."

[0347] This system allows users to receive advance warnings about potential fraud risks, thereby enhancing their personal safety.

[0348] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0349] Step 1:

[0350] The device records ambient sound around the user and converts it into digital data. The input is ambient sound, and the output is digital audio data. Specifically, it collects sound using a built-in microphone and captures the data with audio recording software. An ADC (analog-to-digital converter) is used to convert the recorded analog audio into a digital signal.

[0351] Step 2:

[0352] The terminal sends this digital audio data to the server using a secure communication protocol (e.g., TLS). The input is the digital audio data, and the output is the completion of secure data transmission to the server. Specifically, the terminal encrypts the data and transfers it to the server via the transmission protocol.

[0353] Step 3:

[0354] The server receives transmitted audio data and stores it in a database. The input is digital audio data from the terminal, and the output is the stored database entry. The server processes the received data and stores it in a high-speed database management system.

[0355] Step 4:

[0356] The server runs acoustic analysis software to analyze the stored audio data and extract the basic characteristics of the sound. The input is the stored digital audio data, and the output is the result of the audio analysis. Specifically, the server calculates the spectral characteristics and basic sound patterns of the audio.

[0357] Step 5:

[0358] The server uses an emotion engine to identify emotional information from the tone, speed, and volume of speech. The input is the result of speech analysis, and the output is emotional information. Specifically, it analyzes emotions using machine learning models (e.g., TensorFlow or PyTorch).

[0359] Step 6:

[0360] The server assesses the likelihood of fraud based on extracted sentiment information and issues warnings as necessary. Inputs are sentiment information and keywords from the audio, while output is a warning notification. If the server determines a high risk, it generates an alert and notifies relevant parties via email, SMS, etc.

[0361] (Application Example 2)

[0362] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0363] The problem that this invention aims to solve is to prevent fraud and illegal activities that may occur when using electronic payments by analyzing the emotional state of the user, thereby ensuring security so that transactions can be conducted with peace of mind.

[0364] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0365] In this invention, the server includes means for receiving voice information and storing it in a data storage device, means for analyzing the stored voice information to determine the risk of a transaction, and means for identifying the user's emotional state and confirming the security of the transaction based on that information. This makes it possible to assess the risk of fraud in real time and issue appropriate alerts to the user.

[0366] A "device for collecting audio data" is a terminal that has the function of recording sounds around the user and converting them into digital information.

[0367] "Secure communication" refers to a communication method that uses security protocols to transmit digital information without it being intercepted by a third party.

[0368] An "information processing device" is a computer system that stores received audio information in a database and analyzes that information as needed.

[0369] "Means for determining the risk of a transaction" refers to algorithms and processes that analyze accumulated voice information to evaluate the possibility of fraud or illegal activity.

[0370] A "means for identifying a user's emotional state" refers to a system that has the function of identifying a user's emotions by analyzing tone, speed, intensity, etc., from audio data.

[0371] "Methods for automatically issuing warnings" refers to a system that has the function of quickly sending notifications to users and related parties when a transaction is deemed to be risky.

[0372] The system for carrying out the present invention consists of a device for collecting voice data, a communication means for securely transmitting information, and an information processing device for storing and analyzing voice information. This system is realized by collecting ambient sounds and converting them into digital voice data when a user makes an electronic payment.

[0373] The device that collects voice data operates using an application installed on mobile devices such as smartphones and tablets. It records the user's speech, converts it into digital voice data, and then transmits the data to a server using secure protocols such as SSL and TLS. This application also analyzes emotions from the voice using a generative AI model.

[0374] The server stores the received audio data in a data storage device and simultaneously processes the data using audio analysis technology. Specifically, an AI-powered emotion analysis engine analyzes the tone, speed, and volume of the voice to determine the user's emotional state. Based on this information, the system decides how to act during the electronic payment process. For example, if a user shows anxiety during a transaction, the server immediately evaluates the situation and sends a notification warning of the possibility of fraud.

[0375] This system is used to verify the safety of purchases made on e-commerce sites that users normally use. For example, by providing voice information such as, "Is it really safe to proceed with the purchase of this product?", the system analyzes the information and determines whether it is safe.

[0376] An example of a prompt message for a generative AI model is: "Analyze the voice data and create an alert indicating how to perform security checks on electronic payment transactions if the user's emotional state is anxious." This prompt is used to support real-time feedback functionality based on sentiment analysis.

[0377] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0378] Step 1:

[0379] The device collects ambient sound from the user's surroundings and converts it into digital audio data. The input is ambient sound, which is recorded using a microphone and converted into a digital format output. The converted digital audio data is then prepared for transmission to a server via secure communication.

[0380] Step 2:

[0381] The terminal sends the converted digital audio data to the server using a secure protocol (e.g., SSL / TLS). The input is the digital audio data generated in step 1, and the output arrives securely at the server as encrypted data. This ensures data security and privacy.

[0382] Step 3:

[0383] The server stores the received digital audio data in a database. The input is the encrypted data sent in step 2, the secure communication is decrypted and the digital audio data is converted into an internal format, and the output becomes digital information that is securely stored in the database.

[0384] Step 4:

[0385] The server analyzes the stored audio data. This process includes a generative AI model that uses AI to analyze the tone, speed, and volume of the voice to determine the user's emotional state. The input is the audio data stored in the database in step 3, and the output is information representing the emotional state.

[0386] Step 5:

[0387] The server verifies the security of the transaction based on the analysis results. The input is the sentiment state information obtained in step 4, and by performing a risk assessment according to the sentiment, the server generates an output indicating whether the security of the transaction has been ensured.

[0388] Step 6:

[0389] If the server detects any signs of unease, it will send a warning to the user. The input is the risk assessment result from step 5, and the output is a warning message displayed on the user's terminal. This prompts the user to re-evaluate whether to continue trading.

[0390] Step 7:

[0391] The user reviews the warning displayed on the terminal and reconfirms the security of electronic payments. The input is the warning message received in step 6, and the output prompts the user to take action based on the information needed to determine the specific actions and responses they should take.

[0392] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0393] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0394] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0395] [Third Embodiment]

[0396] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0397] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0398] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0399] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0400] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0401] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0402] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0403] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0404] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0405] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0406] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0407] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0408] To implement this invention, the user first installs a terminal device. The terminal device is placed within the user's living space and configured to continuously monitor everyday conversations at that location. Voice data is collected in real time by the terminal and converted into a digital format. This converted digital data is then transmitted to a server via secure communication.

[0409] The server stores the received audio data in a database and performs audio analysis based on this data. The audio analysis system within the server converts the audio data into text using natural language processing technology and analyzes whether there is a possibility of fraud based on this text. This analysis involves comparing the audio against a database that stores past fraud patterns.

[0410] As a concrete example, suppose a user is having a phone conversation and keywords such as "cash card" and "PIN" are detected consecutively. The server determines that these keywords match a typical pattern of fraud and immediately identifies it as potentially fraudulent. Based on this result, an alert is automatically sent to designated family members or the police, and a notification is also sent to the user's app. This allows the user and their family to take swift action.

[0411] This system can issue real-time alerts for potential fraudulent situations, preventing victims from becoming victims. Since it requires no special operation from the user, it provides simple yet effective fraud prevention.

[0412] The following describes the processing flow.

[0413] Step 1:

[0414] The device constantly monitors the surrounding audio and records it in real time. The recorded audio is immediately converted into digital data.

[0415] Step 2:

[0416] The terminal packages the converted digital data into packets in a batch format at regular intervals and sends them to the server using a secure communication protocol. Encryption technology is used in this process to prevent data leakage and tampering.

[0417] Step 3:

[0418] The server receives the audio data transmitted from the terminal and first stores it in a database. At this stage, the data is tagged with identification tags so that it can be easily retrieved in subsequent analysis.

[0419] Step 4:

[0420] The AI ​​within the server converts the received audio data into text using natural language processing technology. The text data is then passed to an analysis module, which compares it against a database of past fraud patterns.

[0421] Step 5:

[0422] Based on the analysis results, the server activates the alert generation module if it determines that there is a high probability of fraud. The alert information is immediately prepared and passed on to the subsequent alert generation step.

[0423] Step 6:

[0424] The server automatically sends an alert to the relevant family members or police contacts. This alert includes information about the content and date of the suspected fraudulent conversation, as well as specific details about the likelihood of fraud.

[0425] Step 7:

[0426] Users will be notified of potential fraud through apps on their smartphones, etc. Upon receiving a notification, users can check the situation, seek support from family or the police if necessary, and take prompt action against the fraud.

[0427] (Example 1)

[0428] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0429] In modern society, fraudulent activities exploiting communication methods are on the rise, with a particularly high number of incidents targeting the elderly and those vulnerable to information overload. Such fraud poses a serious threat to personal property and safety, making early detection and prevention crucial. However, conventional measures struggle to detect fraudulent activities in real time and respond quickly, resulting in numerous victims. To address this problem, an effective and efficient fraud detection and warning system is necessary.

[0430] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0431] In this invention, the server includes means for converting audio signals into text data using natural language processing technology, means for determining the possibility of fraudulent activity by comparing it with existing information storage, and means for automatically issuing a warning if there is a possibility of fraudulent activity. This makes it possible to analyze the possibility of fraud in real time and issue a warning immediately.

[0432] "Audio signals" refer to information obtained by converting sound vibrations emitted by the environment or people into an electrical format.

[0433] A "communication terminal" is a device used to transmit voice signals to external devices or systems.

[0434] A "digital format" refers to a format in which analog information is represented using a combination of binary numbers that can be processed by a computer.

[0435] "Protective communication technology" refers to communication technology that uses encryption technology and other methods to protect data from interception and tampering by third parties during transmission and reception.

[0436] An "information processing device" refers to a device that has the ability to analyze, store, or transmit received data.

[0437] A "storage device" refers to hardware or digital media used to store information for extended periods.

[0438] "Fraudulent activity" refers to the act of intentionally providing false information with the intent to deceive others and thereby illegally obtaining some kind of benefit.

[0439] "Natural language processing technology" refers to the techniques and algorithms that enable computers to understand and process human language.

[0440] "Text data" refers to the form of strings or sentences obtained by converting audio information into a human-readable format.

[0441] "Information storage" refers to devices or services for storing and managing data.

[0442] A "warning" refers to a notification or alert intended to inform someone of some kind of danger or caution.

[0443] To implement this invention, the user first needs to install a communication terminal in their living space, such as their home or office. The terminal is a device capable of continuously collecting ambient audio signals and converting them into a digital format. By using audio signal processing hardware installed in the terminal, such as a DSP (Digital Signal Processor), analog audio is converted into digital data.

[0444] The converted digital data is transmitted from the terminal to the server using secure communication technology, such as SSL / TLS encryption. The server stores the received data in its information processing unit and prepares it for subsequent analysis. This analysis uses methods to convert speech into text data using natural language processing technology. Specifically, speech recognition software and APIs (Application Programming Interfaces) are used.

[0445] The server compares the obtained text data with existing fraud pattern information. Through this comparison, it is possible to identify potentially fraudulent conversation content. For detected fraudulent activity, the server automatically issues a warning and sends a notification to the user's smartphone app or designated contacts. This process allows users to prevent potential fraud.

[0446] As a concrete example, consider a scenario where a user is receiving a suspicious phone call and their device identifies keywords such as "bank" and "PIN." In such a situation, the server would determine that these keyword patterns are potentially fraudulent and immediately issue a warning. This would allow the user and their family to take swift action based on the notification.

[0447] An example of a prompt statement that utilizes a generative AI model is the question, "Please explain in detail the process from collecting voice data in a living space to detecting fraud." By using this prompt statement, the AI ​​can generate a detailed process explanation, providing useful information for system operation.

[0448] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0449] Step 1:

[0450] The user installs a communication terminal in their living space. The terminal uses a high-sensitivity microphone to collect ambient audio signals in real time. Inputs include ambient sounds and human conversations. The terminal uses an audio signal processing device to convert these audio signals into a digital format and generate digital data as output.

[0451] Step 2:

[0452] The terminal transmits the generated digital data to the server using secure communication technology. Specifically, the digital data is encrypted using the SSL / TLS protocol. The input is digital audio data, and the output is encrypted data that the server can receive.

[0453] Step 3:

[0454] The server decodes the digital data received from the terminal and stores it in the information processing device. It receives encrypted data as input, decodes it, and then produces audio data that can be stored in the database as output.

[0455] Step 4:

[0456] The server converts stored audio data into text using natural language processing technology. It takes audio data as input, generates character data via a speech recognition engine, and outputs accurately transcribed text.

[0457] Step 5:

[0458] The server analyzes the converted text data by comparing it to existing fraud patterns. It takes text data as input and uses topic modeling and keyword matching algorithms to determine the likelihood of fraud. If fraud is highly likely, it provides a flag indicating this.

[0459] Step 6:

[0460] The server issues a warning if it determines that something is potentially fraudulent. It receives a fraud warning flag as input and sends a message to the user's smartphone app or registered contacts via its notification system. As output, it generates an automatically sent warning notification.

[0461] (Application Example 1)

[0462] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0463] In modern society, fraud is becoming increasingly sophisticated, especially fraudulent activities conducted through verbal communication. To prevent such fraud, an effective system is needed that monitors audio in real time and immediately detects potential fraud. Furthermore, if potential fraud is detected, a mechanism is needed to quickly notify those involved and minimize the damage.

[0464] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0465] In this invention, the server includes means for acquiring voice information and converting it into digital information, means for transmitting the digital information using information-protected communication, and means for analyzing the voice information to determine the possibility of fraud and automatically sending a notification. This makes it possible to detect the possibility of fraud with high accuracy and speed and notify relevant parties in real time.

[0466] "Audio information" refers to data obtained by converting ambient sounds into a digital format.

[0467] An "information processing device" is a device that acquires sound and processes it as digital data.

[0468] "Digital information" refers to data in which audio is represented in a digital format.

[0469] "Information protection communication" refers to a communication method for securely transmitting digital data.

[0470] A "processing unit" is a device that analyzes received audio information and stores it on a recording medium.

[0471] A "recording medium" is a physical or electronic medium used to store data.

[0472] "Natural language processing" is a technique for converting audio data into text and analyzing it.

[0473] "Potential fraud" refers to signs that the observed audio data may be related to fraudulent activity.

[0474] "Notification" refers to the act of sending a warning to a user or relevant party when certain conditions are met.

[0475] A "communication device" is a device that analyzes voice data in real time and transmits the results.

[0476] In implementing this invention, the following system configuration is used. A terminal device is placed in the user's surrounding environment and acquires voice information through a microphone. This voice is converted into digital data and transmitted to a server via secure communication. In this process, the Google Cloud Speech-to-Text API is used to convert the voice to text. The server uses natural language processing to analyze the received text data and determine whether it contains fraud keywords. At this stage, the Hugging Face Transformers library is used.

[0477] If a potential scam is detected, the server uses the Twilio API to send a notification to the user, their family, and related parties. This notification allows the user to be aware of the potential scam in real time and take immediate action.

[0478] For example, if a user receives a phone call and keywords such as "PIN" or "bank account number" are uttered during the conversation, the system will immediately identify this as a sign of fraud and send a notification. This process is performed automatically and does not require any special interface interaction from the user.

[0479] The following sentences may be used as prompts for generative AI models:

[0480] Please provide a list of phrases commonly used in scams.

[0481] "How can you detect signs of fraud from everyday conversations?"

[0482] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0483] Step 1:

[0484] The device uses a microphone to acquire ambient audio information. The input audio data is an analog signal, which is then converted into digital data. Specifically, the audio waveform is digitized by sampling via audio recording software.

[0485] Step 2:

[0486] The terminal transmits digital data to the server using secure communication. The input is the digital voice data obtained in the previous step, which is encrypted before transmission. It is common practice to use the standard SSL / TLS protocol to ensure data security.

[0487] Step 3:

[0488] The server converts received digital audio data into text data using the Google Cloud Speech-to-Text API. The input is encrypted digital audio data, speech recognition is performed via the API, and text data is generated as output. Specifically, this involves analyzing the audio waveform and phoneme recognition.

[0489] Step 4:

[0490] The server analyzes the generated text data using natural language processing methods. In this step, the Hugging Face Transformers library is used to evaluate whether the input text data contains fraud-related keywords. The specific actions performed here are matching the text with word patterns and statistical analysis.

[0491] Step 5:

[0492] The server uses the Twilio API to send notifications when it detects potential fraud. The input is the text data that was determined to be highly fraudulent in the previous step, and the output is a warning notification sent to the user or designated parties. The notification can be sent via SMS or email.

[0493] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0494] This invention provides a system that incorporates an emotion engine to recognize emotions from user voice data, thereby achieving advanced fraud prevention. First, the terminal records the sounds around the user and converts them into digital data. The converted voice data is transmitted to the server via secure communication. The server stores the voice data in a database and extracts emotional information using the emotion engine along with basic voice analysis.

[0495] This emotion engine analyzes the tone, speed, and volume of the voice to identify the user's emotional state (e.g., anxiety, anger, relief). Based on this, a system is built to accurately determine the likelihood of fraud. For example, if a user emphasizes keywords such as "urgent" or "important" and also indicates anxiety, the server evaluates this as a high risk of fraud and increases the urgency of the alert.

[0496] For example, if a user is asked over the phone to "provide details of their cash card immediately" and is feeling anxious, the emotion engine will detect this anxiety. The server will then take this anxiety into consideration and send a particularly urgent alert to family members or the police. The notification will include information that the user is showing a stronger-than-usual emotional reaction, prompting a quick response.

[0497] In this way, a system that incorporates an emotion engine allows users to receive feedback on their own emotional state while simultaneously strengthening their crime prevention measures. This system can provide advanced support for preventing fraud and further ensure user safety.

[0498] The following describes the processing flow.

[0499] Step 1:

[0500] The device monitors the surrounding audio in real time and starts recording. The recorded audio is immediately converted into digital data. This digital data is then prepared to be sent to the server via secure communication.

[0501] Step 2:

[0502] The server stores the received audio data in a database. The data is saved in an identifiable format and used for subsequent analysis.

[0503] Step 3:

[0504] An analysis module on the server converts the audio data into text using natural language processing technology, and based on this information, it begins detecting basic fraud patterns. Simultaneously, an emotion engine analyzes the audio data and identifies the user's emotional state from information such as tone and speed of speech.

[0505] Step 4:

[0506] The server detects suspicious interactions by comparing them with a database of fraud patterns and transcribed data. It also considers the results of sentiment analysis and reassessss the risk of fraud if the user is showing anxiety or other strong emotions. This process calculates a fraud likelihood score and determines the urgency of the situation.

[0507] Step 5:

[0508] If a fraudulent activity is deemed highly likely, the server generates an alert, taking into account the results of sentiment analysis. The content and priority of the alert are set according to the urgency and the user's emotional state.

[0509] Step 6:

[0510] The server sends the generated alerts to pre-designated family members or police contacts. The notifications include information about the potential for fraud detected, the user's emotional response, and emergency actions to take.

[0511] Step 7:

[0512] Users receive alert notifications through a smartphone app. These notifications include specific actions to take and, if necessary, contact information for support, enabling quick action.

[0513] (Example 2)

[0514] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0515] In modern society, voice-based fraud is on the rise, and effective preventative measures are needed. However, conventional systems have struggled to analyze emotional information with sufficient accuracy to detect fraudulent activity. Furthermore, they are unable to provide real-time responses based on the user's emotional state, making it impossible to issue effective warnings before fraud is committed.

[0516] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0517] In this invention, the server includes an information processing device means for receiving voice data and storing it in a database, a means for analyzing the intonation, speed, and intensity of the voice and extracting emotional information, and a means for determining the possibility of fraud based on the emotional information and issuing a warning. This enables high-precision evaluation of the risk of fraud experienced by the user through voice and prompt issuance of warnings.

[0518] "Audio data" refers to information used to record and process audio in a digital format.

[0519] A "terminal device" is a device that has the function of recording audio and converting it into digital data.

[0520] "Secure communication" is a technology that provides encrypted and safe communication paths to prevent third parties from eavesdropping on or tampering with data.

[0521] An "information processing device" is a device used to process digital data using a computer.

[0522] A "database" is a system that efficiently stores and manages large amounts of digital information in a searchable format.

[0523] "Voice intonation, speed, and volume" refers to the pitch, speed, and volume of speech, and are elements used to identify emotions.

[0524] "Emotional information" refers to data that indicates a person's emotional state, analyzed from audio data.

[0525] "Potential fraud" refers to the likelihood of fraudulent activity occurring, as determined based on audio data and emotional information.

[0526] A "warning" is an alert sent to users or related parties when a situation is deemed highly likely to be fraudulent.

[0527] This invention is a system that extracts emotional information from a user's voice and determines the likelihood of fraud with high accuracy. This system mainly consists of a terminal device and a server.

[0528] The device records the sounds around the user and converts the recorded audio into digital data. This is done using a built-in microphone and audio recording software. The converted digital data is sent to the server via a secure communication protocol. TLS is a suitable communication protocol for this purpose.

[0529] The server stores the received audio data in a database and performs basic analysis using audio analysis software. Subsequently, the server uses an emotion engine to analyze the intonation, speed, and volume of the speech, extracting the user's emotional information. This emotional information is then analyzed using a machine learning model, which is based on TensorFlow or PyTorch.

[0530] When assessing fraud risk, the server analyzes extracted sentiment information and combines specific keywords with emotional states to determine the likelihood of fraud. Based on this assessment, a warning is generated according to the urgency of the situation and notified family members and relevant parties. Notification methods include email, SMS, and push notifications via a dedicated app.

[0531] For example, if a user calls and says, "I want to know the details of my cash card," and an anxious feeling is detected, the emotion engine will detect that anxiety. The server will then determine this to be high risk and immediately issue a warning.

[0532] An example of a prompt for a generative AI model is, "Explain how many steps are taken to process the audio data and how the alert is ultimately generated."

[0533] This system allows users to receive advance warnings about potential fraud risks, thereby enhancing their personal safety.

[0534] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0535] Step 1:

[0536] The device records ambient sound around the user and converts it into digital data. The input is ambient sound, and the output is digital audio data. Specifically, it collects sound using a built-in microphone and captures the data with audio recording software. An ADC (analog-to-digital converter) is used to convert the recorded analog audio into a digital signal.

[0537] Step 2:

[0538] The terminal sends this digital audio data to the server using a secure communication protocol (e.g., TLS). The input is the digital audio data, and the output is the completion of secure data transmission to the server. Specifically, the terminal encrypts the data and transfers it to the server via the transmission protocol.

[0539] Step 3:

[0540] The server receives transmitted audio data and stores it in a database. The input is digital audio data from the terminal, and the output is the stored database entry. The server processes the received data and stores it in a high-speed database management system.

[0541] Step 4:

[0542] The server runs acoustic analysis software to analyze the stored audio data and extract the basic characteristics of the sound. The input is the stored digital audio data, and the output is the result of the audio analysis. Specifically, the server calculates the spectral characteristics and basic sound patterns of the audio.

[0543] Step 5:

[0544] The server uses an emotion engine to identify emotional information from the tone, speed, and volume of speech. The input is the result of speech analysis, and the output is emotional information. Specifically, it analyzes emotions using machine learning models (e.g., TensorFlow or PyTorch).

[0545] Step 6:

[0546] The server assesses the likelihood of fraud based on extracted sentiment information and issues warnings as necessary. Inputs are sentiment information and keywords from the audio, while output is a warning notification. If the server determines a high risk, it generates an alert and notifies relevant parties via email, SMS, etc.

[0547] (Application Example 2)

[0548] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0549] The problem that this invention aims to solve is to prevent fraud and illegal activities that may occur when using electronic payments by analyzing the emotional state of the user, thereby ensuring security so that transactions can be conducted with peace of mind.

[0550] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0551] In this invention, the server includes means for receiving voice information and storing it in a data storage device, means for analyzing the stored voice information to determine the risk of a transaction, and means for identifying the user's emotional state and confirming the security of the transaction based on that information. This makes it possible to assess the risk of fraud in real time and issue appropriate alerts to the user.

[0552] A "device for collecting audio data" is a terminal that has the function of recording sounds around the user and converting them into digital information.

[0553] "Secure communication" refers to a communication method that uses security protocols to transmit digital information without it being intercepted by a third party.

[0554] An "information processing device" is a computer system that stores received audio information in a database and analyzes that information as needed.

[0555] "Means for determining the risk of a transaction" refers to algorithms and processes that analyze accumulated voice information to evaluate the possibility of fraud or illegal activity.

[0556] A "means for identifying a user's emotional state" refers to a system that has the function of identifying a user's emotions by analyzing tone, speed, intensity, etc., from audio data.

[0557] "Methods for automatically issuing warnings" refers to a system that has the function of quickly sending notifications to users and related parties when a transaction is deemed to be risky.

[0558] The system for carrying out the present invention consists of a device for collecting voice data, a communication means for securely transmitting information, and an information processing device for storing and analyzing voice information. This system is realized by collecting ambient sounds and converting them into digital voice data when a user makes an electronic payment.

[0559] The device that collects voice data operates using an application installed on mobile devices such as smartphones and tablets. It records the user's speech, converts it into digital voice data, and then transmits the data to a server using secure protocols such as SSL and TLS. This application also analyzes emotions from the voice using a generative AI model.

[0560] The server stores the received audio data in a data storage device and simultaneously processes the data using audio analysis technology. Specifically, an AI-powered emotion analysis engine analyzes the tone, speed, and volume of the voice to determine the user's emotional state. Based on this information, the system decides how to act during the electronic payment process. For example, if a user shows anxiety during a transaction, the server immediately evaluates the situation and sends a notification warning of the possibility of fraud.

[0561] This system is used to verify the safety of purchases made on e-commerce sites that users normally use. For example, by providing voice information such as, "Is it really safe to proceed with the purchase of this product?", the system analyzes the information and determines whether it is safe.

[0562] An example of a prompt message for a generative AI model is: "Analyze the voice data and create an alert indicating how to perform security checks on electronic payment transactions if the user's emotional state is anxious." This prompt is used to support real-time feedback functionality based on sentiment analysis.

[0563] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0564] Step 1:

[0565] The device collects ambient sound from the user's surroundings and converts it into digital audio data. The input is ambient sound, which is recorded using a microphone and converted into a digital format output. The converted digital audio data is then prepared for transmission to a server via secure communication.

[0566] Step 2:

[0567] The terminal sends the converted digital audio data to the server using a secure protocol (e.g., SSL / TLS). The input is the digital audio data generated in step 1, and the output arrives securely at the server as encrypted data. This ensures data security and privacy.

[0568] Step 3:

[0569] The server stores the received digital audio data in a database. The input is the encrypted data sent in step 2, the secure communication is decrypted and the digital audio data is converted into an internal format, and the output becomes digital information that is securely stored in the database.

[0570] Step 4:

[0571] The server analyzes the stored audio data. This process includes a generative AI model that uses AI to analyze the tone, speed, and volume of the voice to determine the user's emotional state. The input is the audio data stored in the database in step 3, and the output is information representing the emotional state.

[0572] Step 5:

[0573] The server verifies the security of the transaction based on the analysis results. The input is the sentiment state information obtained in step 4, and by performing a risk assessment according to the sentiment, the server generates an output indicating whether the security of the transaction has been ensured.

[0574] Step 6:

[0575] If the server detects any signs of unease, it will send a warning to the user. The input is the risk assessment result from step 5, and the output is a warning message displayed on the user's terminal. This prompts the user to re-evaluate whether to continue trading.

[0576] Step 7:

[0577] The user reviews the warning displayed on the terminal and reconfirms the security of electronic payments. The input is the warning message received in step 6, and the output prompts the user to take action based on the information needed to determine the specific actions and responses they should take.

[0578] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0579] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0580] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0581] [Fourth Embodiment]

[0582] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0583] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0584] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0585] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0586] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0587] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0588] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0589] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0590] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0591] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0592] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0593] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0594] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0595] To implement this invention, the user first installs a terminal device. The terminal device is placed within the user's living space and configured to continuously monitor everyday conversations at that location. Voice data is collected in real time by the terminal and converted into a digital format. This converted digital data is then transmitted to a server via secure communication.

[0596] The server stores the received audio data in a database and performs audio analysis based on this data. The audio analysis system within the server converts the audio data into text using natural language processing technology and analyzes whether there is a possibility of fraud based on this text. This analysis involves comparing the audio against a database that stores past fraud patterns.

[0597] As a concrete example, suppose a user is having a phone conversation and keywords such as "cash card" and "PIN" are detected consecutively. The server determines that these keywords match a typical pattern of fraud and immediately identifies it as potentially fraudulent. Based on this result, an alert is automatically sent to designated family members or the police, and a notification is also sent to the user's app. This allows the user and their family to take swift action.

[0598] This system can issue real-time alerts for potential fraudulent situations, preventing victims from becoming victims. Since it requires no special operation from the user, it provides simple yet effective fraud prevention.

[0599] The following describes the processing flow.

[0600] Step 1:

[0601] The device constantly monitors the surrounding audio and records it in real time. The recorded audio is immediately converted into digital data.

[0602] Step 2:

[0603] The terminal packages the converted digital data into packets in a batch format at regular intervals and sends them to the server using a secure communication protocol. Encryption technology is used in this process to prevent data leakage and tampering.

[0604] Step 3:

[0605] The server receives the audio data transmitted from the terminal and first stores it in a database. At this stage, the data is tagged with identification tags so that it can be easily retrieved in subsequent analysis.

[0606] Step 4:

[0607] The AI ​​within the server converts the received audio data into text using natural language processing technology. The text data is then passed to an analysis module, which compares it against a database of past fraud patterns.

[0608] Step 5:

[0609] Based on the analysis results, the server activates the alert generation module if it determines that there is a high probability of fraud. The alert information is immediately prepared and passed on to the subsequent alert generation step.

[0610] Step 6:

[0611] The server automatically sends an alert to the relevant family members or police contacts. This alert includes information about the content and date of the suspected fraudulent conversation, as well as specific details about the likelihood of fraud.

[0612] Step 7:

[0613] Users will be notified of potential fraud through apps on their smartphones, etc. Upon receiving a notification, users can check the situation, seek support from family or the police if necessary, and take prompt action against the fraud.

[0614] (Example 1)

[0615] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0616] In modern society, fraudulent activities exploiting communication methods are on the rise, with a particularly high number of incidents targeting the elderly and those vulnerable to information overload. Such fraud poses a serious threat to personal property and safety, making early detection and prevention crucial. However, conventional measures struggle to detect fraudulent activities in real time and respond quickly, resulting in numerous victims. To address this problem, an effective and efficient fraud detection and warning system is necessary.

[0617] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0618] In this invention, the server includes means for converting audio signals into text data using natural language processing technology, means for determining the possibility of fraudulent activity by comparing it with existing information storage, and means for automatically issuing a warning if there is a possibility of fraudulent activity. This makes it possible to analyze the possibility of fraud in real time and issue a warning immediately.

[0619] "Audio signals" refer to information obtained by converting sound vibrations emitted by the environment or people into an electrical format.

[0620] A "communication terminal" is a device used to transmit voice signals to external devices or systems.

[0621] A "digital format" refers to a format in which analog information is represented using a combination of binary numbers that can be processed by a computer.

[0622] "Protective communication technology" refers to communication technology that uses encryption technology and other methods to protect data from interception and tampering by third parties during transmission and reception.

[0623] An "information processing device" refers to a device that has the ability to analyze, store, or transmit received data.

[0624] A "storage device" refers to hardware or digital media used to store information for extended periods.

[0625] "Fraudulent activity" refers to the act of intentionally providing false information with the intent to deceive others and thereby illegally obtaining some kind of benefit.

[0626] "Natural language processing technology" refers to the techniques and algorithms that enable computers to understand and process human language.

[0627] "Text data" refers to the form of strings or sentences obtained by converting audio information into a human-readable format.

[0628] "Information storage" refers to devices or services for storing and managing data.

[0629] A "warning" refers to a notification or alert intended to inform someone of some kind of danger or caution.

[0630] To implement this invention, the user first needs to install a communication terminal in their living space, such as their home or office. The terminal is a device capable of continuously collecting ambient audio signals and converting them into a digital format. By using audio signal processing hardware installed in the terminal, such as a DSP (Digital Signal Processor), analog audio is converted into digital data.

[0631] The converted digital data is transmitted from the terminal to the server using secure communication technology, such as SSL / TLS encryption. The server stores the received data in its information processing unit and prepares it for subsequent analysis. This analysis uses methods to convert speech into text data using natural language processing technology. Specifically, speech recognition software and APIs (Application Programming Interfaces) are used.

[0632] The server compares the obtained text data with existing fraud pattern information. Through this comparison, it is possible to identify potentially fraudulent conversation content. For detected fraudulent activity, the server automatically issues a warning and sends a notification to the user's smartphone app or designated contacts. This process allows users to prevent potential fraud.

[0633] As a concrete example, consider a scenario where a user is receiving a suspicious phone call and their device identifies keywords such as "bank" and "PIN." In such a situation, the server would determine that these keyword patterns are potentially fraudulent and immediately issue a warning. This would allow the user and their family to take swift action based on the notification.

[0634] An example of a prompt statement that utilizes a generative AI model is the question, "Please explain in detail the process from collecting voice data in a living space to detecting fraud." By using this prompt statement, the AI ​​can generate a detailed process explanation, providing useful information for system operation.

[0635] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0636] Step 1:

[0637] The user installs a communication terminal in their living space. The terminal uses a high-sensitivity microphone to collect ambient audio signals in real time. Inputs include ambient sounds and human conversations. The terminal uses an audio signal processing device to convert these audio signals into a digital format and generate digital data as output.

[0638] Step 2:

[0639] The terminal transmits the generated digital data to the server using secure communication technology. Specifically, the digital data is encrypted using the SSL / TLS protocol. The input is digital audio data, and the output is encrypted data that the server can receive.

[0640] Step 3:

[0641] The server decodes the digital data received from the terminal and stores it in the information processing device. It receives encrypted data as input, decodes it, and then produces audio data that can be stored in the database as output.

[0642] Step 4:

[0643] The server converts stored audio data into text using natural language processing technology. It takes audio data as input, generates character data via a speech recognition engine, and outputs accurately transcribed text.

[0644] Step 5:

[0645] The server analyzes the converted text data by comparing it to existing fraud patterns. It takes text data as input and uses topic modeling and keyword matching algorithms to determine the likelihood of fraud. If fraud is highly likely, it provides a flag indicating this.

[0646] Step 6:

[0647] The server issues a warning if it determines that something is potentially fraudulent. It receives a fraud warning flag as input and sends a message to the user's smartphone app or registered contacts via its notification system. As output, it generates an automatically sent warning notification.

[0648] (Application Example 1)

[0649] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0650] In modern society, fraud is becoming increasingly sophisticated, especially fraudulent activities conducted through verbal communication. To prevent such fraud, an effective system is needed that monitors audio in real time and immediately detects potential fraud. Furthermore, if potential fraud is detected, a mechanism is needed to quickly notify those involved and minimize the damage.

[0651] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0652] In this invention, the server includes means for acquiring voice information and converting it into digital information, means for transmitting the digital information using information-protected communication, and means for analyzing the voice information to determine the possibility of fraud and automatically sending a notification. This makes it possible to detect the possibility of fraud with high accuracy and speed and notify relevant parties in real time.

[0653] "Audio information" refers to data obtained by converting ambient sounds into a digital format.

[0654] An "information processing device" is a device that acquires sound and processes it as digital data.

[0655] "Digital information" refers to data in which audio is represented in a digital format.

[0656] "Information protection communication" refers to a communication method for securely transmitting digital data.

[0657] A "processing unit" is a device that analyzes received audio information and stores it on a recording medium.

[0658] A "recording medium" is a physical or electronic medium used to store data.

[0659] "Natural language processing" is a technique for converting audio data into text and analyzing it.

[0660] "Potential fraud" refers to signs that the observed audio data may be related to fraudulent activity.

[0661] "Notification" refers to the act of sending a warning to a user or relevant party when certain conditions are met.

[0662] A "communication device" is a device that analyzes voice data in real time and transmits the results.

[0663] In implementing this invention, the following system configuration is used. A terminal device is placed in the user's surrounding environment and acquires voice information through a microphone. This voice is converted into digital data and transmitted to a server via secure communication. In this process, the Google Cloud Speech-to-Text API is used to convert the voice to text. The server uses natural language processing to analyze the received text data and determine whether it contains fraud keywords. At this stage, the Hugging Face Transformers library is used.

[0664] If a potential scam is detected, the server uses the Twilio API to send a notification to the user, their family, and related parties. This notification allows the user to be aware of the potential scam in real time and take immediate action.

[0665] For example, if a user receives a phone call and keywords such as "PIN" or "bank account number" are uttered during the conversation, the system will immediately identify this as a sign of fraud and send a notification. This process is performed automatically and does not require any special interface interaction from the user.

[0666] The following sentences may be used as prompts for generative AI models:

[0667] Please provide a list of phrases commonly used in scams.

[0668] "How can you detect signs of fraud from everyday conversations?"

[0669] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0670] Step 1:

[0671] The device uses a microphone to acquire ambient audio information. The input audio data is an analog signal, which is then converted into digital data. Specifically, the audio waveform is digitized by sampling via audio recording software.

[0672] Step 2:

[0673] The terminal transmits digital data to the server using secure communication. The input is the digital voice data obtained in the previous step, which is encrypted before transmission. It is common practice to use the standard SSL / TLS protocol to ensure data security.

[0674] Step 3:

[0675] The server converts received digital audio data into text data using the Google Cloud Speech-to-Text API. The input is encrypted digital audio data, speech recognition is performed via the API, and text data is generated as output. Specifically, this involves analyzing the audio waveform and phoneme recognition.

[0676] Step 4:

[0677] The server analyzes the generated text data using natural language processing methods. In this step, the Hugging Face Transformers library is used to evaluate whether the input text data contains fraud-related keywords. The specific actions performed here are matching the text with word patterns and statistical analysis.

[0678] Step 5:

[0679] The server uses the Twilio API to send notifications when it detects potential fraud. The input is the text data that was determined to be highly fraudulent in the previous step, and the output is a warning notification sent to the user or designated parties. The notification can be sent via SMS or email.

[0680] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0681] This invention provides a system that incorporates an emotion engine to recognize emotions from user voice data, thereby achieving advanced fraud prevention. First, the terminal records the sounds around the user and converts them into digital data. The converted voice data is transmitted to the server via secure communication. The server stores the voice data in a database and extracts emotional information using the emotion engine along with basic voice analysis.

[0682] This emotion engine analyzes the tone, speed, and volume of the voice to identify the user's emotional state (e.g., anxiety, anger, relief). Based on this, a system is built to accurately determine the likelihood of fraud. For example, if a user emphasizes keywords such as "urgent" or "important" and also indicates anxiety, the server evaluates this as a high risk of fraud and increases the urgency of the alert.

[0683] For example, if a user is asked over the phone to "provide details of their cash card immediately" and is feeling anxious, the emotion engine will detect this anxiety. The server will then take this anxiety into consideration and send a particularly urgent alert to family members or the police. The notification will include information that the user is showing a stronger-than-usual emotional reaction, prompting a quick response.

[0684] In this way, a system that incorporates an emotion engine allows users to receive feedback on their own emotional state while simultaneously strengthening their crime prevention measures. This system can provide advanced support for preventing fraud and further ensure user safety.

[0685] The following describes the processing flow.

[0686] Step 1:

[0687] The device monitors the surrounding audio in real time and starts recording. The recorded audio is immediately converted into digital data. This digital data is then prepared to be sent to the server via secure communication.

[0688] Step 2:

[0689] The server stores the received audio data in a database. The data is saved in an identifiable format and used for subsequent analysis.

[0690] Step 3:

[0691] An analysis module on the server converts the audio data into text using natural language processing technology, and based on this information, it begins detecting basic fraud patterns. Simultaneously, an emotion engine analyzes the audio data and identifies the user's emotional state from information such as tone and speed of speech.

[0692] Step 4:

[0693] The server detects suspicious interactions by comparing them with a database of fraud patterns and transcribed data. It also considers the results of sentiment analysis and reassessss the risk of fraud if the user is showing anxiety or other strong emotions. This process calculates a fraud likelihood score and determines the urgency of the situation.

[0694] Step 5:

[0695] If a fraudulent activity is deemed highly likely, the server generates an alert, taking into account the results of sentiment analysis. The content and priority of the alert are set according to the urgency and the user's emotional state.

[0696] Step 6:

[0697] The server sends the generated alerts to pre-designated family members or police contacts. The notifications include information about the potential for fraud detected, the user's emotional response, and emergency actions to take.

[0698] Step 7:

[0699] Users receive alert notifications through a smartphone app. These notifications include specific actions to take and, if necessary, contact information for support, enabling quick action.

[0700] (Example 2)

[0701] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0702] In modern society, voice-based fraud is on the rise, and effective preventative measures are needed. However, conventional systems have struggled to analyze emotional information with sufficient accuracy to detect fraudulent activity. Furthermore, they are unable to provide real-time responses based on the user's emotional state, making it impossible to issue effective warnings before fraud is committed.

[0703] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0704] In this invention, the server includes an information processing device means for receiving voice data and storing it in a database, a means for analyzing the intonation, speed, and intensity of the voice and extracting emotional information, and a means for determining the possibility of fraud based on the emotional information and issuing a warning. This enables high-precision evaluation of the risk of fraud experienced by the user through voice and prompt issuance of warnings.

[0705] "Audio data" refers to information used to record and process audio in a digital format.

[0706] A "terminal device" is a device that has the function of recording audio and converting it into digital data.

[0707] "Secure communication" is a technology that provides encrypted and safe communication paths to prevent third parties from eavesdropping on or tampering with data.

[0708] An "information processing device" is a device used to process digital data using a computer.

[0709] A "database" is a system that efficiently stores and manages large amounts of digital information in a searchable format.

[0710] "Voice intonation, speed, and volume" refers to the pitch, speed, and volume of speech, and are elements used to identify emotions.

[0711] "Emotional information" refers to data that indicates a person's emotional state, analyzed from audio data.

[0712] "Potential fraud" refers to the likelihood of fraudulent activity occurring, as determined based on audio data and emotional information.

[0713] A "warning" is an alert sent to users or related parties when a situation is deemed highly likely to be fraudulent.

[0714] This invention is a system that extracts emotional information from a user's voice and determines the likelihood of fraud with high accuracy. This system mainly consists of a terminal device and a server.

[0715] The device records the sounds around the user and converts the recorded audio into digital data. This is done using a built-in microphone and audio recording software. The converted digital data is sent to the server via a secure communication protocol. TLS is a suitable communication protocol for this purpose.

[0716] The server stores the received audio data in a database and performs basic analysis using audio analysis software. Subsequently, the server uses an emotion engine to analyze the intonation, speed, and volume of the speech, extracting the user's emotional information. This emotional information is then analyzed using a machine learning model, which is based on TensorFlow or PyTorch.

[0717] When assessing fraud risk, the server analyzes extracted sentiment information and combines specific keywords with emotional states to determine the likelihood of fraud. Based on this assessment, a warning is generated according to the urgency of the situation and notified family members and relevant parties. Notification methods include email, SMS, and push notifications via a dedicated app.

[0718] For example, if a user calls and says, "I want to know the details of my cash card," and an anxious feeling is detected, the emotion engine will detect that anxiety. The server will then determine this to be high risk and immediately issue a warning.

[0719] An example of a prompt for a generative AI model is, "Explain how many steps are taken to process the audio data and how the alert is ultimately generated."

[0720] This system allows users to receive advance warnings about potential fraud risks, thereby enhancing their personal safety.

[0721] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0722] Step 1:

[0723] The device records ambient sound around the user and converts it into digital data. The input is ambient sound, and the output is digital audio data. Specifically, it collects sound using a built-in microphone and captures the data with audio recording software. An ADC (analog-to-digital converter) is used to convert the recorded analog audio into a digital signal.

[0724] Step 2:

[0725] The terminal sends this digital audio data to the server using a secure communication protocol (e.g., TLS). The input is the digital audio data, and the output is the completion of secure data transmission to the server. Specifically, the terminal encrypts the data and transfers it to the server via the transmission protocol.

[0726] Step 3:

[0727] The server receives transmitted audio data and stores it in a database. The input is digital audio data from the terminal, and the output is the stored database entry. The server processes the received data and stores it in a high-speed database management system.

[0728] Step 4:

[0729] The server runs acoustic analysis software to analyze the stored audio data and extract the basic characteristics of the sound. The input is the stored digital audio data, and the output is the result of the audio analysis. Specifically, the server calculates the spectral characteristics and basic sound patterns of the audio.

[0730] Step 5:

[0731] The server uses an emotion engine to identify emotional information from the tone, speed, and volume of speech. The input is the result of speech analysis, and the output is emotional information. Specifically, it analyzes emotions using machine learning models (e.g., TensorFlow or PyTorch).

[0732] Step 6:

[0733] The server assesses the likelihood of fraud based on extracted sentiment information and issues warnings as necessary. Inputs are sentiment information and keywords from the audio, while output is a warning notification. If the server determines a high risk, it generates an alert and notifies relevant parties via email, SMS, etc.

[0734] (Application Example 2)

[0735] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0736] The problem that this invention aims to solve is to prevent fraud and illegal activities that may occur when using electronic payments by analyzing the emotional state of the user, thereby ensuring security so that transactions can be conducted with peace of mind.

[0737] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0738] In this invention, the server includes means for receiving voice information and storing it in a data storage device, means for analyzing the stored voice information to determine the risk of a transaction, and means for identifying the user's emotional state and confirming the security of the transaction based on that information. This makes it possible to assess the risk of fraud in real time and issue appropriate alerts to the user.

[0739] A "device for collecting audio data" is a terminal that has the function of recording sounds around the user and converting them into digital information.

[0740] "Secure communication" refers to a communication method that uses security protocols to transmit digital information without it being intercepted by a third party.

[0741] An "information processing device" is a computer system that stores received audio information in a database and analyzes that information as needed.

[0742] "Means for determining the risk of a transaction" refers to algorithms and processes that analyze accumulated voice information to evaluate the possibility of fraud or illegal activity.

[0743] A "means for identifying a user's emotional state" refers to a system that has the function of identifying a user's emotions by analyzing tone, speed, intensity, etc., from audio data.

[0744] "Methods for automatically issuing warnings" refers to a system that has the function of quickly sending notifications to users and related parties when a transaction is deemed to be risky.

[0745] The system for carrying out the present invention consists of a device for collecting voice data, a communication means for securely transmitting information, and an information processing device for storing and analyzing voice information. This system is realized by collecting ambient sounds and converting them into digital voice data when a user makes an electronic payment.

[0746] The device that collects voice data operates using an application installed on mobile devices such as smartphones and tablets. It records the user's speech, converts it into digital voice data, and then transmits the data to a server using secure protocols such as SSL and TLS. This application also analyzes emotions from the voice using a generative AI model.

[0747] The server stores the received audio data in a data storage device and simultaneously processes the data using audio analysis technology. Specifically, an AI-powered emotion analysis engine analyzes the tone, speed, and volume of the voice to determine the user's emotional state. Based on this information, the system decides how to act during the electronic payment process. For example, if a user shows anxiety during a transaction, the server immediately evaluates the situation and sends a notification warning of the possibility of fraud.

[0748] This system is used to verify the safety of purchases made on e-commerce sites that users normally use. For example, by providing voice information such as, "Is it really safe to proceed with the purchase of this product?", the system analyzes the information and determines whether it is safe.

[0749] An example of a prompt message for a generative AI model is: "Analyze the voice data and create an alert indicating how to perform security checks on electronic payment transactions if the user's emotional state is anxious." This prompt is used to support real-time feedback functionality based on sentiment analysis.

[0750] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0751] Step 1:

[0752] The device collects ambient sound from the user's surroundings and converts it into digital audio data. The input is ambient sound, which is recorded using a microphone and converted into a digital format output. The converted digital audio data is then prepared for transmission to a server via secure communication.

[0753] Step 2:

[0754] The terminal sends the converted digital audio data to the server using a secure protocol (e.g., SSL / TLS). The input is the digital audio data generated in step 1, and the output arrives securely at the server as encrypted data. This ensures data security and privacy.

[0755] Step 3:

[0756] The server stores the received digital audio data in a database. The input is the encrypted data sent in step 2, the secure communication is decrypted and the digital audio data is converted into an internal format, and the output becomes digital information that is securely stored in the database.

[0757] Step 4:

[0758] The server analyzes the stored audio data. This process includes a generative AI model that uses AI to analyze the tone, speed, and volume of the voice to determine the user's emotional state. The input is the audio data stored in the database in step 3, and the output is information representing the emotional state.

[0759] Step 5:

[0760] The server verifies the security of the transaction based on the analysis results. The input is the sentiment state information obtained in step 4, and by performing a risk assessment according to the sentiment, the server generates an output indicating whether the security of the transaction has been ensured.

[0761] Step 6:

[0762] If the server detects any signs of unease, it will send a warning to the user. The input is the risk assessment result from step 5, and the output is a warning message displayed on the user's terminal. This prompts the user to re-evaluate whether to continue trading.

[0763] Step 7:

[0764] The user reviews the warning displayed on the terminal and reconfirms the security of electronic payments. The input is the warning message received in step 6, and the output prompts the user to take action based on the information needed to determine the specific actions and responses they should take.

[0765] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0766] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0767] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0768] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0769] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0770] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0771] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0772] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0773] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0774] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0775] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0776] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0777] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0778] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0779] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0780] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0781] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0782] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0783] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0784] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0785] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted as being incorporated by reference.

[0786] The following is further disclosed regarding the embodiments described above.

[0787] (Claim 1)

[0788] A terminal device for collecting audio data, comprising means for recording ambient sounds and converting them into digital data,

[0789] A means for transmitting the aforementioned digital data using secure communication,

[0790] A server device that receives the transmitted audio data and stores it in a database,

[0791] A means for analyzing the aforementioned accumulated audio data to determine the possibility of fraud,

[0792] A method to automatically send out alerts in case of potential fraud,

[0793] A system that includes this.

[0794] (Claim 2)

[0795] The system according to claim 1, comprising means for converting audio data into text using natural language processing technology.

[0796] (Claim 3)

[0797] The system according to claim 1, comprising means for determining the possibility of fraud by comparing it with an existing database.

[0798] "Example 1"

[0799] (Claim 1)

[0800] A communication terminal for collecting audio signals, comprising means for recording ambient sounds and converting them into a digital format,

[0801] A means for transmitting the data converted to the aforementioned digital format using protected communication technology,

[0802] An information processing device that receives the transmitted audio signal and stores it in a storage device,

[0803] A means for analyzing the stored audio signal to determine the possibility of fraudulent activity,

[0804] A method for automatically issuing warnings in the event of potential fraud,

[0805] A system that includes this.

[0806] (Claim 2)

[0807] The system according to claim 1, comprising means for converting an audio signal into text data using natural language processing technology.

[0808] (Claim 3)

[0809] The system according to claim 1, comprising means for determining the possibility of fraudulent activity by comparing it with existing information storage.

[0810] "Application Example 1"

[0811] (Claim 1)

[0812] An information processing device for acquiring audio information, comprising means for recording ambient sound and converting it into digital information,

[0813] A configuration means for transmitting the aforementioned digital information using information protection communication,

[0814] A computing device means that receives the transmitted audio information and stores it in a recording medium,

[0815] A configuration means for analyzing the stored audio information to determine the possibility of fraud,

[0816] A configuration means for automatically sending a notification when a potential fraud is detected,

[0817] A configuration means for controlling communication equipment for analyzing voice and determining danger in real time,

[0818] A system that includes this.

[0819] (Claim 2)

[0820] A system according to claim 1, which converts audio information into written form using natural language processing.

[0821] (Claim 3)

[0822] The system according to claim 1, which determines the possibility of fraud by matching it with pre-stored record information.

[0823] "Example 2 of combining an emotion engine"

[0824] (Claim 1)

[0825] A terminal device for collecting audio data, comprising means for recording ambient sounds and converting them into digital data,

[0826] A means for transmitting the aforementioned digital data using secure communication,

[0827] An information processing device that receives the transmitted audio data and stores it in a database,

[0828] A means for analyzing the accumulated audio data and extracting emotional information by analyzing the intonation, speed, and intensity of the voice,

[0829] A means of accurately determining the possibility of fraud based on emotional information and issuing a warning if fraud is suspected,

[0830] A system that includes this.

[0831] (Claim 2)

[0832] The system according to claim 1, comprising means for converting audio data into text using natural language processing technology.

[0833] (Claim 3)

[0834] The system according to claim 1, comprising means for determining the possibility of fraud by comparing it with an existing information database.

[0835] "Application example 2 when combining with an emotional engine"

[0836] (Claim 1)

[0837] A device for collecting audio data, comprising means for recording ambient sounds and converting them into digital information,

[0838] Means for transmitting the aforementioned digital information using secure communication,

[0839] An information processing device that receives the transmitted audio information and stores it in a data storage device,

[0840] A means for analyzing the accumulated audio information to determine the risk of the transaction,

[0841] A means of identifying the user's emotional state and verifying the security of transactions based on that information,

[0842] A means of automatically issuing a warning when there is a risk,

[0843] A system that includes this.

[0844] (Claim 2)

[0845] The system according to claim 1, comprising means for converting speech information into text data using natural language processing technology.

[0846] (Claim 3)

[0847] The system according to claim 1, comprising means for determining the risk of a transaction by comparing it with existing information. [Explanation of symbols]

[0848] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A terminal device for collecting audio data, comprising means for recording ambient sounds and converting them into digital data, A means for transmitting the aforementioned digital data using secure communication, A server device that receives the transmitted audio data and stores it in a database, A means for analyzing the aforementioned accumulated audio data to determine the possibility of fraud, A method to automatically send out alerts in case of potential fraud, A system that includes this.

2. The system according to claim 1, comprising means for converting audio data into text using natural language processing technology.

3. The system according to claim 1, comprising means for determining the possibility of fraud by comparing it with an existing database.

Images