system

Abstract

We provide the system. [Solution] A means for collecting acoustic data to continuously acquire user voice data, A data transmission means for securely transferring the aforementioned audio data to the cloud, An artificial intelligence analysis means for analyzing the aforementioned audio data and evaluating the possibility of fraud, An alert generation method for generating a warning when the risk of fraud exceeds a certain threshold, A notification means for sending the aforementioned alarm to a pre-registered communication destination, A system that includes this.

Description

Technical Field

[0004] , , ,

[0005] , , ,

[0001] The technology of the present disclosure relates to a system.

Background Art

[0002] Patent Document 1 discloses a method for controlling a persona chatbot, which is performed by at least one processor and includes steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to an explanation of a character of the chatbot, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Deposit fraud targeting the elderly has become a social problem, and the damage is increasing year by year. However, there is a problem that effective means for preventing the damage are limited. In particular, it is often difficult for the elderly themselves to recognize that it is fraud, and post - incident response is the main focus. For this reason, a system that detects signs of fraud in real time and issues a warning in advance is required.

Means for Solving the Problems

[0005] This invention provides a system that continuously collects user voice data, securely transfers it to the cloud, and analyzes it. Voice data is acquired by an acoustic data collection means and transferred to the cloud by a data transmission means. This data is analyzed in real time by an artificial intelligence analysis means, and the likelihood of fraud is evaluated by comparing it with a database of past fraud cases. If the fraud risk exceeds a certain threshold, an alert is generated by an alert generation means, and the alert is sent to a pre-registered communication destination using a notification means. This makes it possible to quickly detect signs of fraud and prevent damage.

[0006] "Audio data acquisition means" refers to devices or software used to acquire sounds from the user's surroundings.

[0007] A "data transmission method" is a system that has the function of securely transferring collected audio data to a cloud server.

[0008] "Artificial intelligence analysis means" refers to AI technology that has the function of analyzing voice data and evaluating the possibility of fraud.

[0009] An "alert generation system" is a system equipped with the function to generate an alert when a risk of fraud is identified.

[0010] A "notification means" refers to a function or system for sending generated alarms to pre-registered communication destinations.

[0011] A "database of past fraud cases" is a database that stores information about fraud cases that have occurred in the past. [Brief explanation of the drawing]

[0012] [Figure 1] This is a conceptual diagram showing an example of the configuration of a data processing system according to the first embodiment. [Figure 2] This is a conceptual diagram showing an example of the essential functions of a data processing device and a smart device according to the first embodiment. [Figure 3] It is a conceptual diagram showing an example of the configuration of a data processing system according to the second embodiment. [Figure 4] It is a conceptual diagram showing an example of the main functions of a data processing device and smart glasses according to the second embodiment. [Figure 5] It is a conceptual diagram showing an example of the configuration of a data processing system according to the third embodiment. [Figure 6] It is a conceptual diagram showing an example of the main functions of a data processing device and a headset-type terminal according to the third embodiment. [Figure 7] It is a conceptual diagram showing an example of the configuration of a data processing system according to the fourth embodiment. [Figure 8] It is a conceptual diagram showing an example of the main functions of a data processing device and a robot according to the fourth embodiment. [Figure 9] It shows an emotion map to which a plurality of emotions are mapped. [Figure 10] It shows an emotion map to which a plurality of emotions are mapped. [Figure 11] It is a sequence diagram showing the processing flow of the data processing system in Example 1. [Figure 12] It is a sequence diagram showing the processing flow of the data processing system in Application Example 1. [Figure 13] It is a sequence diagram showing the processing flow of the data processing system in Example 2 when an emotion engine is combined. [Figure 14] It is a sequence diagram showing the processing flow of the data processing system in Application Example 2 when an emotion engine is combined.

Mode for Carrying Out the Invention

[0013] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described according to the accompanying drawings. .

[0014] First, the language used in the following description will be explained.

[0015] In the following embodiments, the numbered processor (hereinafter simply referred to as "processor") may be a single arithmetic unit or a combination of multiple arithmetic units. Also, the processor may be a single type of arithmetic unit or a combination of multiple types of arithmetic units. Examples of arithmetic units include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), and the like.

[0016] In the following embodiments, the numbered RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a work memory by the processor.

[0017] In the following embodiments, the numbered storage is one or more non-volatile storage devices that store various programs and various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), or magnetic tapes, etc.

[0018] In the following embodiments, the numbered communication I / F (Interface) is an interface including a communication processor and an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), or Bluetooth (registered trademark), etc. <000……此处数字不全,无法完整翻译

[0019] ,请补充完整后继续提问。[[ID=……此处数字不全,无法完整翻译 ,请补充完整后继续提问。In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." That is, "A and / or B" means that it may be A alone, or B alone, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" applies when expressing three or more things linked by "and / or."

[0020] [First Embodiment]

[0021] Figure 1 shows an example of the configuration of the data processing system 10 according to the first embodiment.

[0022] As shown in Figure 1, the data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0023] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0024] The smart device 14 comprises a computer 36, a reception device 38, an output device 40, a camera 42, and a communication interface 44. The computer 36 comprises a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The reception device 38, output device 40, and camera 42 are also connected to the bus 52.

[0025] The reception device 38 is equipped with a touch panel 38A and a microphone 38B, etc., and receives user input. The touch panel 38A receives user input by detecting contact with an object (e.g., a pen or finger). The microphone 38B receives user input by detecting the user's voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0026] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form perceptible to the user 20 (e.g., audio and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs audio according to instructions from the processor 46. The camera 42 is a small digital camera equipped with an optical system such as a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0027] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various types of information between processor 46 and processor 28 via network 54.

[0028] Figure 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0029] As shown in Figure 2, in the data processing device 12, a specific processing is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" related to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 according to the specific processing program 56 executed on the RAM 30.

[0030] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0031] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The reception output program 60 is used in conjunction with a specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0032] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0033] This invention is implemented as a voice analysis system to prevent deposit fraud targeting the elderly. This system continuously acquires the user's voice, analyzes it in real time, and functions to detect signs of fraud early and issue a warning.

[0034] Users first need either a device equipped with acoustic data collection capabilities or a smartphone with a dedicated application installed. This device has the function of collecting ambient sounds using a microphone, encrypting the data within the device, and temporarily storing it. The collected audio data is then transferred to a cloud server using a secure protocol.

[0035] The server processes the received audio data using its built-in artificial intelligence analysis capabilities to accurately assess the likelihood of fraud. The analysis involves transcribing the audio into text and comparing it with a database of past fraud cases to identify keywords and conversational patterns specific to fraud. When signs of fraud are detected, the server generates a warning using an alert generation mechanism and quickly sends this information to the user's device.

[0036] The device receives the warning and alerts the user with a push notification and an alert sound. Simultaneously, a notification is sent to pre-set emergency contacts to encourage a quick response. For example, if a user receives a suspicious investment solicitation over the phone, the audio is analyzed in real time, and if it is determined to be potentially fraudulent, an automatic notification is sent to family members or the police immediately.

[0037] In this way, this system provides users with peace of mind by constantly monitoring the risk of fraud lurking in everyday communication and responding quickly.

[0038] The following describes the processing flow.

[0039] Step 1:

[0040] The device constantly collects sounds from the user's surroundings using a microphone and captures them as audio data. This audio data is encrypted for privacy protection and temporarily stored on the device.

[0041] Step 2:

[0042] The terminal transfers pre-processed audio data to the cloud server using a secure protocol. During this process, the data remains encrypted and protected from tampering.

[0043] Step 3:

[0044] The server receives the audio data sent to the cloud and analyzes the audio using artificial intelligence analysis tools. This analysis converts the audio into text and detects keywords and conversation patterns specific to fraud.

[0045] Step 4:

[0046] The server compares detected keywords and patterns with a database of past fraud cases to assess the risk of fraud. Based on this assessment, it determines whether the case is highly likely to be fraudulent.

[0047] Step 5:

[0048] The server generates an alert using an alert generation mechanism if it determines that the risk of fraud exceeds a certain threshold. This alert includes the fraud assessment results and a text sample of the audio.

[0049] Step 6:

[0050] The server sends the generated alarm to the user's device and pre-registered emergency contacts. Upon receiving the alarm, the user's device is notified via push notification or an alert sound.

[0051] Step 7:

[0052] Users receive warnings from their devices and are prompted to take appropriate action. This allows users and stakeholders to take swift action to prevent fraud.

[0053] (Example 1)

[0054] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0055] Fraudulent activities targeting the elderly are becoming more sophisticated year by year, and there is a need for effective means to prevent them from becoming victims. However, conventional methods make it difficult to detect fraud in real time and provide rapid notification, and there are also issues of user privacy. The present invention aims to solve these problems and provide a system that can detect the possibility of fraud early with high accuracy and quickly notify users and relevant parties of that information.

[0056] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0057] In this invention, the server includes machine learning analysis means for analyzing voice information, means for converting voice to text using natural language processing technology, and warning generation means for generating alarms and providing notifications. This enables real-time analysis of voice information, allowing for highly accurate detection of signs of fraud and the rapid and secure transmission of warnings to relevant parties.

[0058] "Audio information acquisition means" refers to a device or technology that continuously collects sounds from the user's surroundings.

[0059] "Information transmission means" refers to protocols and devices for securely transferring acquired voice information over a communication network.

[0060] "Machine learning analysis methods" refer to techniques that use artificial intelligence technology to analyze voice information and assess the likelihood of fraud.

[0061] A "warning generation method" is a technology that automatically generates an alarm when the risk of fraud exceeds a certain threshold.

[0062] A "notification means" refers to a method or device for transmitting a generated alarm to a pre-registered communication destination.

[0063] "Natural language processing technology" is a technology for converting spoken information into text and analyzing its content.

[0064] An "information display means" is an interface for visually or audibly communicating warnings or notifications to the user.

[0065] This invention relates to a voice analysis system for protecting the elderly from fraud. The system achieves its specific functions through the cooperation of a user, a terminal, and a server. The user uses a mobile terminal with a dedicated application installed to collect voice information. This terminal is equipped with hardware and software to acquire ambient sounds with a microphone, encrypt the data, and temporarily store it.

[0066] The device transmits the acquired voice information to a server in the cloud using a secure protocol. HTTPS is often used as the protocol. The server converts the voice to text and compares it with a database containing past fraud cases. This process utilizes speech recognition and natural language processing technologies.

[0067] The server incorporates a generative AI model to analyze voice information and detect fraudulent keywords and conversation patterns. If it determines that there is a high probability of fraud, the server immediately generates an alarm and notifies the device of this information. The notification is conveyed to the user through the device's push notification function or visual alert.

[0068] For example, when a user receives a suspicious phone solicitation, the system analyzes the conversation in real time and, if it determines the content to be fraudulent, immediately generates an alert and notifies family members and the police. This protects users from fraud in real time.

[0069] Furthermore, as an example of prompts to the generating AI model, questions such as "How can AI be used to issue real-time warnings if an elderly person is about to become a victim of bank deposit fraud?" can be asked. This system will protect users from the risk of fraud and enable them to live a safer life.

[0070] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0071] Step 1:

[0072] The user activates a device equipped with an acoustic information acquisition mechanism and begins collecting audio data. The device uses a microphone to acquire ambient acoustic signals and encrypts this data within the device. The input is the ambient audio signals received by the microphone, and the output is the encrypted audio data. Specifically, when the user launches a dedicated application, the device automatically begins collecting audio.

[0073] Step 2:

[0074] The device transfers encrypted audio data to a server in the cloud using a secure protocol. The input is encrypted audio data stored on the device, and the output is encrypted data sent to the server. Specifically, data transmission uses the HTTPS protocol and is performed in the background without user intervention.

[0075] Step 3:

[0076] The server decrypts the received encrypted audio data and converts it to text using natural language processing techniques. The input is the encrypted audio data that arrives at the server, and the output is the audio information converted into text. Specifically, the server runs a speech recognition algorithm to accurately convert the audio into text.

[0077] Step 4:

[0078] The server analyzes transcribed audio data using machine learning analysis tools to assess the risk of fraud. The input is transcribed audio information, and the output is the fraud risk assessment result. Specifically, the server analyzes keywords and conversation patterns specific to fraud and scores the likelihood of fraud using a generative AI model.

[0079] Step 5:

[0080] If the server determines that there is a high probability of fraud, it immediately generates an alarm and notifies the terminal of this information. The input is the result of the fraud risk assessment, and the output is the generated alarm message. Specifically, the server creates the warning message in real time and sends it to the terminal via the notification system.

[0081] Step 6:

[0082] The device notifies the user of received alarms via push notifications and alert sounds. The input is the alarm message sent from the server, and the output is the warning notification to the user. Specifically, the device displays the warning message on the screen and simultaneously sounds an alert to draw the user's attention.

[0083] This allows the system to detect signs of fraud with high accuracy in real time, supporting a swift response.

[0084] (Application Example 1)

[0085] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart device 14 will be referred to as the "terminal."

[0086] To address the vulnerability of the elderly to fraudulent verbal transactions such as bank deposit scams, technology is needed that analyzes voices in real time, quickly detects signs of fraud, and implements appropriate protective measures. Furthermore, there is a need for a system that is simple and automated for the elderly, providing a sense of security and safety.

[0087] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0088] In this invention, the server includes a voice acquisition mechanism for continuously acquiring user voice information, an information transmission mechanism for securely transferring voice information to a cloud infrastructure, and a machine learning analysis mechanism for evaluating the likelihood of fraud. This makes it possible to quickly detect signs of fraud, generate an alarm, and immediately notify the user and registered emergency contacts.

[0089] A "voice acquisition mechanism" is a device that plays the role of continuously collecting the user's voice information.

[0090] An "information transmission mechanism" is a device that transfers collected voice information to a cloud infrastructure via a secure communication protocol.

[0091] A "machine learning analysis mechanism" is a device that uses artificial intelligence technology to analyze collected audio information and evaluate the likelihood of fraud.

[0092] An "alarm generation mechanism" is a device that generates an alarm when the risk of fraud exceeds a certain threshold.

[0093] A "notification transmission mechanism" is a device that transmits generated alarms to pre-registered communication destinations.

[0094] An "instant notification mechanism" is a device that instantly notifies the user's mobile device or communication partner of an alarm.

[0095] An "information aggregation platform" is a database system that stores data on past fraud cases to help determine the likelihood of fraud.

[0096] The system realizing this invention begins with a terminal equipped with a voice acquisition mechanism that the user wears or carries. The terminal continuously collects ambient sound using a highly sensitive microphone and securely transmits this voice information to a cloud platform. In the cloud platform, the voice information is analyzed by a machine learning analysis mechanism to assess the likelihood of fraud. The analysis uses speech recognition technologies such as Google® Cloud Speech-to-Text API and a generative AI model that has learned the characteristics of fraud.

[0097] Based on the analysis results, the server generates an alert if the risk of fraud exceeds a certain threshold. When an alert is generated, it is sent to pre-registered contacts via a notification system. In addition, an instant notification system immediately notifies the user's mobile device, promptly alerting the user and their family.

[0098] For example, if a user receives a suspicious investment solicitation over the phone, the system collects the audio recording. If the recording is identified as containing dangerous keywords such as "Please transfer the money immediately," the system quickly generates an alert to inform the user of the situation. An alert is also immediately sent to pre-registered emergency contacts, such as family members' smartphones, allowing them to take immediate action to protect the user.

[0099] An example of a prompt for a generative AI model would be: "Analyze this voice conversation to see if it contains signs of fraud. The text is: 'Please provide your account number now...'"

[0100] This system makes it possible to prevent the risk of fraud targeting the elderly and provide a safe and secure living environment.

[0101] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0102] Step 1:

[0103] When the device is powered on, it uses its voice acquisition mechanism to collect ambient sounds. The input is ambient sound, and the output is audio data converted into a digital format. This data is temporarily stored within the device.

[0104] Step 2:

[0105] The device encrypts the collected audio data and transfers it to the cloud infrastructure using a secure protocol (e.g., HTTPS). The input is the encrypted audio data, and the output is the completion of the transmission to the cloud infrastructure.

[0106] Step 3:

[0107] The server receives audio data via a cloud infrastructure and performs analysis using a machine learning analysis mechanism. The input is the audio data received by the server, and the output is the transcribed conversation content and the results of the fraud probability assessment. For data processing, speech recognition technology (e.g., Google Cloud Speech-to-Text API) is used to convert speech to text, and then a generative AI model performs a fraud risk assessment.

[0108] Step 4:

[0109] Based on the analysis results, the server generates an alert if the fraud risk exceeds a certain threshold. The input is the assessment result of the likelihood of fraud, and the output is the trigger for the warning notification. Specifically, the alarm generation mechanism generates an alert when the risk determination threshold is exceeded.

[0110] Step 5:

[0111] The server utilizes a notification transmission mechanism to send generated alarms to pre-registered communication destinations and the user's mobile device. Inputs are the alarm notification and communication destination information, while output is the transmission of the alarm to recipients. Specifically, the immediate notification mechanism pushes the alarm to the user and simultaneously sends an alert to emergency contacts.

[0112] This series of processes makes it possible to detect fraud risks in real time and respond quickly.

[0113] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0114] This invention aims to improve the accuracy of fraud risk assessment by combining an emotion engine with a system that analyzes user voice data in real time and evaluates the likelihood of fraud. In addition to the conventional functions of collecting and analyzing user voice data, this system further improves fraud risk assessment and alert generation by recognizing the user's emotional state.

[0115] Users first use a smartphone with a dedicated app installed or a device equipped with a dedicated audio data collection system. The device continuously collects audio from the user's surroundings, encrypts it, and temporarily stores it. This audio data is then sent to a cloud server as needed.

[0116] Upon receiving audio data, the server first converts the audio into text using artificial intelligence analysis tools to detect keywords and patterns related to fraud. Next, it uses an emotion engine to extract the user's emotional state from the audio data. This emotional information is considered as one factor in assessing fraud risk.

[0117] For example, if the emotion engine determines that a user is in an excited state, it can prioritize the alert, considering the fraud risk to be higher than usual. In this way, emotional states are used as supplementary information to adjust the fraud risk assessment.

[0118] If the server determines that the situation is high-risk, it generates an alert using an alert generation mechanism and promptly notifies the user's device and pre-configured emergency contacts via a notification mechanism. By receiving the warning on their device, the user can take swift action to prevent fraud.

[0119] In this way, by incorporating an emotion engine, we can improve the accuracy of detecting fraudulent activity and provide a system that enhances user safety.

[0120] The following describes the processing flow.

[0121] Step 1:

[0122] The device constantly collects audio from the user's surroundings using its microphone. This audio data is encrypted for privacy protection and temporarily stored on the device.

[0123] Step 2:

[0124] The device encrypts the collected audio data and transfers it to the cloud server using a secure protocol. This ensures data confidentiality and prevents tampering.

[0125] Step 3:

[0126] Before analyzing the received audio data, the server converts it into text. This conversion process uses speech recognition technology.

[0127] Step 4:

[0128] The server processes the converted text using artificial intelligence analysis to detect keywords and patterns related to fraud. The detected information is used for an initial assessment of fraud risk.

[0129] Step 5:

[0130] The server uses an emotion engine to extract the user's emotional state from the audio data. Specifically, it analyzes the tone of voice and other factors to determine whether the user is excited, tense, or calm.

[0131] Step 6:

[0132] The server integrates the fraud assessment from step 4 and the emotional state from step 5 to reassess the fraud risk. If the emotional state is associated with a high risk, such as excitement, the risk assessment is adjusted.

[0133] Step 7:

[0134] If the server determines that there is a high risk of fraud, it will generate an alert using an alert generation mechanism. The alert will include the results of the fraud risk assessment and information on the emotional state.

[0135] Step 8:

[0136] The server quickly sends the generated alarm to the user's device and registered emergency contacts. Upon receiving the alarm, the device alerts the user using push notifications and an alert sound.

[0137] Step 9:

[0138] Users receive warnings from their devices and take appropriate action as needed. This allows them to take appropriate measures to prevent becoming a victim of fraud.

[0139] (Example 2)

[0140] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart device 14 as the "terminal".

[0141] Fraudulent activities are becoming increasingly sophisticated, making detection extremely difficult. In particular, fraudsters employ psychological tricks to undermine victims' vigilance, limiting the effectiveness of traditional keyword-based detection methods. Furthermore, monitoring systems that fail to adequately consider users' emotional states can lead to inaccurate risk assessments. Therefore, there is a need for more accurate fraud risk assessment systems.

[0142] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0143] In this invention, the server includes acoustic information collection means for continuously acquiring the user's voice information, information transmission means for securely transferring the voice information to an information processing device, and emotion analysis means for extracting emotional states from the voice information and including emotional information in the fraud risk assessment. This makes it possible to assess fraud risk with greater accuracy while taking into account the user's emotional state.

[0144] "User" refers to an individual or group that uses the system.

[0145] "Audio information" refers to the user's speech and surrounding sounds collected as acoustic data.

[0146] "Acoustic information acquisition means" refers to a device or method used to acquire the user's voice information.

[0147] An "information processing device" refers to a computer system used for data analysis, storage, and transfer.

[0148] "Information transmission means" refers to technical means for securely transferring collected audio information to a server or cloud.

[0149] "Machine learning analysis methods" refer to AI technologies and algorithms used to analyze voice information and assess the likelihood of fraud.

[0150] "Emotional analysis means" refers to a technology or algorithm for extracting the user's emotional state from voice information.

[0151] "Alert generation means" refers to a function that generates warnings or alerts when the risk of fraud exceeds a certain threshold.

[0152] "Notification means" refers to a communication means used to transmit generated alarms to users or pre-registered communication partners.

[0153] This invention combines a sentiment analysis function with a fraud risk assessment system that analyzes voice data in real time to enhance user safety. Specific embodiments are described below.

[0154] First, the user uses a mobile device or acoustic information collection device with a dedicated application installed that can collect voice information. The device uses a built-in microphone to continuously acquire the user's voice information and temporarily stores it as digital data. The data is protected from unauthorized access by others using encryption methods such as AES.

[0155] After the terminal collects voice information, it securely transmits the encrypted voice information to an information processing device in the cloud using an information transmission method. Communication is conducted via the HTTPS protocol, guaranteeing data integrity and confidentiality.

[0156] The server analyzes the received data. First, it uses machine learning analysis to convert the audio data into text data. For general text conversion, natural language processing models, known as speech recognition technology, are used. Furthermore, the server uses sentiment analysis to extract the user's emotional state from the audio data. Sentiment analysis employs algorithms that analyze the tone and rhythm of the voice.

[0157] If the server determines that there is a high risk of fraud, it will notify the user via an alarm generation system. This includes push notifications to the device and email notifications to registered communication partners. This allows users to take immediate action.

[0158] For example, when a user suddenly starts using a particular word frequently, emotional analysis can detect tension or excitement. As a result, the risk of fraud is assessed as higher than usual. An example of a prompt message that might be input to the generating AI model is, "Analyze the emotional signs related to fraud from this audio data."

[0159] By incorporating sentiment analysis in this way, we can provide a system that offers more accurate fraud risk assessments and supports users' safer lives.

[0160] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0161] Step 1:

[0162] The terminal acquires audio information captured by the acoustic information collection device. The microphone detects all sounds the user makes and captures them as digital audio data. At this point, the input is an analog audio signal, and the output is digital audio data. The digitized audio information is temporarily stored on the terminal. During this time, the data is protected by AES encryption.

[0163] Step 2:

[0164] The terminal uses an information transmission method to send encrypted voice information to a server in the cloud. The input here is encrypted digital voice data, and the output is data securely transferred to the server. The HTTPS protocol ensures data integrity and confidentiality.

[0165] Step 3:

[0166] The server uses machine learning analysis to convert audio information into text. It receives digital audio data as input and outputs text data. By utilizing an acoustic model to convert audio into text, it creates a format suitable for analysis.

[0167] Step 4:

[0168] The server uses emotion analysis tools to analyze transcribed data and voice tone to extract the user's emotional state. The inputs are text data and digital voice data, while the output is data indicating the emotional state. The emotion analysis model detects excitement and tension based on the intonation and speed of the voice.

[0169] Step 5:

[0170] The server assesses the risk of fraud based on the analyzed data. The input consists of keyword detection results and sentiment information, while the output is a fraud risk assessment index. This assessment uses a generative AI model to determine the risk according to the prompt "Assess the fraud risk from this audio data."

[0171] Step 6:

[0172] The server issues an alert using an alarm generation mechanism if it determines that the fraud risk is high. The input is a fraud risk assessment index, and the output is the generated alarm. The alarm is pushed to the device and, in some cases, also notified to emergency contacts.

[0173] This series of processes allows users to respond quickly to fraud, improving security.

[0174] (Application Example 2)

[0175] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as a "server" and the smart device 14 as a "terminal".

[0176] In modern society, while online payments and voice-based transactions are increasing, fraudulent activities are also becoming more sophisticated. Therefore, there is a growing need for systems that can analyze voice data in real time and assess fraud risk with greater accuracy. In particular, there is a demand for systems that can interpret emotions from voice and assess fraud risk while taking the user's psychological state into account, but achieving this presents significant challenges.

[0177] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0178] In this invention, the server includes acoustic data collection means for continuously acquiring user voice information, data transmission means for encrypting the voice information and transferring it to a remote computer, artificial intelligence analysis means for converting the voice information into text and detecting words and patterns related to fraud, and an emotion analysis engine for recognizing emotional states and reflecting them in the fraud possibility assessment. This makes it possible to accurately assess the risk of fraud using emotion information extracted from voice data and appropriately notify the user of a warning.

[0179] "Acoustic data acquisition means" refers to devices or programs that continuously acquire ambient sound information, and have the function of collecting sound via smartphones or microphones.

[0180] "Data transmission means" refers to technologies and methods for encrypting collected audio information and securely transferring it to a remote computer.

[0181] "Artificial intelligence analysis means" refers to technologies and algorithms used to convert audio information into text and detect words and patterns related to fraud from that text.

[0182] A "sentiment analysis engine" refers to analytical technology that recognizes a user's emotional state based on voice information and reflects it in relation to the assessment of the risk of fraud.

[0183] An "alert generation method" refers to a device or method that generates an alert and notifies the user or registered contacts when the risk level of fraud exceeds a predetermined threshold.

[0184] "Notification method" refers to the technology or method used to send generated alarms to users or pre-registered contacts.

[0185] This invention is realized by equipping the user's terminal with acoustic data collection means. The terminal constantly collects audio from the user's surroundings, encrypts it, and temporarily stores it. This encrypted audio information is transferred to a cloud server in a secure manner using data transmission means as needed.

[0186] The server converts received audio information into text using artificial intelligence analysis tools to detect words and patterns related to fraud. Software such as Google Cloud Speech-to-Text and AWS® Transcribe can be used in this process. Furthermore, a sentiment analysis engine is utilized to identify the user's emotional state from the audio information. IBM Watson® Tone Analyzer and Microsoft® Text Analytics are suitable for this analysis.

[0187] The server combines data obtained from emotional states and voice information to assess the risk of fraud. If the risk of fraud exceeds a threshold, an alert generation system creates a warning and quickly sends the warning to the user's device or pre-registered contacts via a notification system.

[0188] For example, if an analysis of the audio generated when a user attempts to purchase an electronic device online detects an abnormal level of excitement, it is determined that the user is likely to be a fraud. In this situation, the user is warned to perform additional verification, such as by being prompted to enable two-factor authentication.

[0189] An example of a prompt is: "Detect whether the user is experiencing emotional distress during online payment and assess the likelihood of fraud."

[0190] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0191] Step 1:

[0192] The device continuously collects audio from the user's surroundings using acoustic data collection equipment. This audio data is acquired in real time as input and temporarily stored within the device. The data is encrypted and prepared for transmission to a cloud server in a secure state.

[0193] Step 2:

[0194] The terminal transfers encrypted voice information to a cloud server using a data transmission method. In this process, data is transferred over the network and prepared for reception by the server. The input is encrypted voice data, and the server receives the data as output.

[0195] Step 3:

[0196] The server uses artificial intelligence analysis to convert the received audio data into text. Here, Google Cloud Speech-to-Text is used as the data processing tool to convert audio data to text. The input to this process is audio data, and the output is text data in which fraud-related terms can be detected.

[0197] Step 4:

[0198] The server uses a sentiment analysis engine to determine the user's emotional state from text data. The input is the text data generated in step 3, and the output is information about the user's emotional state. IBM Watson Tone Analyzer is used for this sentiment assessment.

[0199] Step 5:

[0200] The server assesses the risk of fraud based on fraud-related terms and sentiment detected by artificial intelligence analysis. The input is text and sentiment data related to fraud risk, and the output is a risk assessment. If this risk level exceeds a threshold, an alert generation mechanism is triggered.

[0201] Step 6:

[0202] An alert is generated by the alert generation mechanism. This alert suggests a high probability of fraud and is ready to be sent as output to the user's device and registered contacts.

[0203] Step 7:

[0204] The server uses a notification system to send generated alarms to the user's device and designated contacts. The input is the generated alarm, and the output is a notification sent to the user and contacts. This allows the user to take immediate action.

[0205] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0206] Data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of data generation model 58 is ChatGPT (registered trademark) (Internet search).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0207] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart device 14.

[0208] [Second Embodiment]

[0209] Figure 3 shows an example of the configuration of the data processing system 210 according to the second embodiment.

[0210] As shown in Figure 3, the data processing system 210 includes a data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0211] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0212] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication interface 44. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, and camera 42 are also connected to the bus 52.

[0213] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0214] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0215] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0216] Figure 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Figure 4, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0217] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0218] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0219] In the smart glasses 214, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0220] Next, the identification processing performed by the identification processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0221] This invention is implemented as a voice analysis system to prevent deposit fraud targeting the elderly. This system continuously acquires the user's voice, analyzes it in real time, and functions to detect signs of fraud early and issue a warning.

[0222] Users first need either a device equipped with acoustic data collection capabilities or a smartphone with a dedicated application installed. This device has the function of collecting ambient sounds using a microphone, encrypting the data within the device, and temporarily storing it. The collected audio data is then transferred to a cloud server using a secure protocol.

[0223] The server processes the received audio data using its built-in artificial intelligence analysis capabilities to accurately assess the likelihood of fraud. The analysis involves transcribing the audio into text and comparing it with a database of past fraud cases to identify keywords and conversational patterns specific to fraud. When signs of fraud are detected, the server generates a warning using an alert generation mechanism and quickly sends this information to the user's device.

[0224] The device receives the warning and alerts the user with a push notification and an alert sound. Simultaneously, a notification is sent to pre-set emergency contacts to encourage a quick response. For example, if a user receives a suspicious investment solicitation over the phone, the audio is analyzed in real time, and if it is determined to be potentially fraudulent, an automatic notification is sent to family members or the police immediately.

[0225] In this way, this system provides users with peace of mind by constantly monitoring the risk of fraud lurking in everyday communication and responding quickly.

[0226] The following describes the processing flow.

[0227] Step 1:

[0228] The device constantly collects sounds from the user's surroundings using a microphone and captures them as audio data. This audio data is encrypted for privacy protection and temporarily stored on the device.

[0229] Step 2:

[0230] The terminal transfers pre-processed audio data to the cloud server using a secure protocol. During this process, the data remains encrypted and protected from tampering.

[0231] Step 3:

[0232] The server receives the audio data sent to the cloud and analyzes the audio using artificial intelligence analysis tools. This analysis converts the audio into text and detects keywords and conversation patterns specific to fraud.

[0233] Step 4:

[0234] The server compares detected keywords and patterns with a database of past fraud cases to assess the risk of fraud. Based on this assessment, it determines whether the case is highly likely to be fraudulent.

[0235] Step 5:

[0236] The server generates an alert using an alert generation mechanism if it determines that the risk of fraud exceeds a certain threshold. This alert includes the fraud assessment results and a text sample of the audio.

[0237] Step 6:

[0238] The server sends the generated alarm to the user's device and pre-registered emergency contacts. Upon receiving the alarm, the user's device is notified via push notification or an alert sound.

[0239] Step 7:

[0240] Users receive warnings from their devices and are prompted to take appropriate action. This allows users and stakeholders to take swift action to prevent fraud.

[0241] (Example 1)

[0242] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0243] Fraudulent activities targeting the elderly are becoming more sophisticated year by year, and there is a need for effective means to prevent them from becoming victims. However, conventional methods make it difficult to detect fraud in real time and provide rapid notification, and there are also issues of user privacy. The present invention aims to solve these problems and provide a system that can detect the possibility of fraud early with high accuracy and quickly notify users and relevant parties of that information.

[0244] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0245] In this invention, the server includes machine learning analysis means for analyzing voice information, means for converting voice to text using natural language processing technology, and warning generation means for generating alarms and providing notifications. This enables real-time analysis of voice information, allowing for highly accurate detection of signs of fraud and the rapid and secure transmission of warnings to relevant parties.

[0246] "Audio information acquisition means" refers to a device or technology that continuously collects sounds from the user's surroundings.

[0247] "Information transmission means" refers to protocols and devices for securely transferring acquired voice information over a communication network.

[0248] "Machine learning analysis methods" refer to techniques that use artificial intelligence technology to analyze voice information and assess the likelihood of fraud.

[0249] A "warning generation method" is a technology that automatically generates an alarm when the risk of fraud exceeds a certain threshold.

[0250] A "notification means" refers to a method or device for transmitting a generated alarm to a pre-registered communication destination.

[0251] "Natural language processing technology" is a technology for converting spoken information into text and analyzing its content.

[0252] An "information display means" is an interface for visually or audibly communicating warnings or notifications to the user.

[0253] This invention relates to a voice analysis system for protecting the elderly from fraud. The system achieves its specific functions through the cooperation of a user, a terminal, and a server. The user uses a mobile terminal with a dedicated application installed to collect voice information. This terminal is equipped with hardware and software to acquire ambient sounds with a microphone, encrypt the data, and temporarily store it.

[0254] The device transmits the acquired voice information to a server in the cloud using a secure protocol. HTTPS is often used as the protocol. The server converts the voice to text and compares it with a database containing past fraud cases. This process utilizes speech recognition and natural language processing technologies.

[0255] The server incorporates a generative AI model to analyze voice information and detect fraudulent keywords and conversation patterns. If it determines that there is a high probability of fraud, the server immediately generates an alarm and notifies the device of this information. The notification is conveyed to the user through the device's push notification function or visual alert.

[0256] For example, when a user receives a suspicious phone solicitation, the system analyzes the conversation in real time and, if it determines the content to be fraudulent, immediately generates an alert and notifies family members and the police. This protects users from fraud in real time.

[0257] Furthermore, as an example of prompts to the generating AI model, questions such as "How can AI be used to issue real-time warnings if an elderly person is about to become a victim of bank deposit fraud?" can be asked. This system will protect users from the risk of fraud and enable them to live a safer life.

[0258] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0259] Step 1:

[0260] The user activates a device equipped with an acoustic information acquisition mechanism and begins collecting audio data. The device uses a microphone to acquire ambient acoustic signals and encrypts this data within the device. The input is the ambient audio signals received by the microphone, and the output is the encrypted audio data. Specifically, when the user launches a dedicated application, the device automatically begins collecting audio.

[0261] Step 2:

[0262] The device transfers encrypted audio data to a server in the cloud using a secure protocol. The input is encrypted audio data stored on the device, and the output is encrypted data sent to the server. Specifically, data transmission uses the HTTPS protocol and is performed in the background without user intervention.

[0263] Step 3:

[0264] The server decrypts the received encrypted audio data and converts it to text using natural language processing techniques. The input is the encrypted audio data that arrives at the server, and the output is the audio information converted into text. Specifically, the server runs a speech recognition algorithm to accurately convert the audio into text.

[0265] Step 4:

[0266] The server analyzes transcribed audio data using machine learning analysis tools to assess the risk of fraud. The input is transcribed audio information, and the output is the fraud risk assessment result. Specifically, the server analyzes keywords and conversation patterns specific to fraud and scores the likelihood of fraud using a generative AI model.

[0267] Step 5:

[0268] If the server determines that there is a high probability of fraud, it immediately generates an alarm and notifies the terminal of this information. The input is the result of the fraud risk assessment, and the output is the generated alarm message. Specifically, the server creates the warning message in real time and sends it to the terminal via the notification system.

[0269] Step 6:

[0270] The device notifies the user of received alarms via push notifications and alert sounds. The input is the alarm message sent from the server, and the output is the warning notification to the user. Specifically, the device displays the warning message on the screen and simultaneously sounds an alert to draw the user's attention.

[0271] This allows the system to detect signs of fraud with high accuracy in real time, supporting a swift response.

[0272] (Application Example 1)

[0273] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0274] To address the vulnerability of the elderly to fraudulent verbal transactions such as bank deposit scams, technology is needed that analyzes voices in real time, quickly detects signs of fraud, and implements appropriate protective measures. Furthermore, there is a need for a system that is simple and automated for the elderly, providing a sense of security and safety.

[0275] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0276] In this invention, the server includes a voice acquisition mechanism for continuously acquiring user voice information, an information transmission mechanism for securely transferring voice information to a cloud infrastructure, and a machine learning analysis mechanism for evaluating the likelihood of fraud. This makes it possible to quickly detect signs of fraud, generate an alarm, and immediately notify the user and registered emergency contacts.

[0277] A "voice acquisition mechanism" is a device that plays the role of continuously collecting the user's voice information.

[0278] An "information transmission mechanism" is a device that transfers collected voice information to a cloud infrastructure via a secure communication protocol.

[0279] A "machine learning analysis mechanism" is a device that uses artificial intelligence technology to analyze collected audio information and evaluate the likelihood of fraud.

[0280] An "alarm generation mechanism" is a device that generates an alarm when the risk of fraud exceeds a certain threshold.

[0281] A "notification transmission mechanism" is a device that transmits generated alarms to pre-registered communication destinations.

[0282] An "instant notification mechanism" is a device that instantly notifies the user's mobile device or communication partner of an alarm.

[0283] An "information aggregation platform" is a database system that stores data on past fraud cases to help determine the likelihood of fraud.

[0284] The system for realizing this invention starts from a terminal equipped with an audio acquisition mechanism worn or carried by a user. The terminal constantly collects ambient audio using a high-sensitivity microphone and transfers the audio information to the cloud platform in a secure way. In the cloud platform, the audio information is analyzed by a machine learning analysis mechanism to evaluate the possibility of fraud. For the analysis, voice recognition technologies such as the Google Cloud Speech-to-Text API and a generative AI model that has learned the characteristics of fraud are used.

[0285] Based on the analysis results, the server generates an alarm when the risk of fraud exceeds a certain standard. When an alarm is generated, it is sent to the pre-registered contacts through the notification sending mechanism. Also, through the instant notification mechanism, the alarm is immediately notified to the user's mobile information terminal, quickly prompting the user and their family to pay attention.

[0286] As a specific example, when audio of a user receiving a suspicious investment solicitation over the phone is collected by the system and its content is identified as containing a dangerous keyword such as "Transfer money immediately now", the system quickly generates an alarm and notifies the user of the situation. Also, an alarm is immediately sent to the pre-registered emergency contacts, such as the smartphones of family members, enabling immediate measures to protect the user.

[0287] An example of a prompt sentence for the generative AI model is in the form of "Analyze whether this voice conversation contains signs of fraud. The text is as follows: 'Now immediately account number...'".

[0288] With this system, it is possible to prevent the risk of fraud targeting the elderly in advance and provide a safe and reassuring living environment.

[0289] The flow of the specific process in Application Example 1 will be described using FIG. 12.

[0290] Step 1:

[0291] When the device is powered on, it uses its voice acquisition mechanism to collect ambient sounds. The input is ambient sound, and the output is audio data converted into a digital format. This data is temporarily stored within the device.

[0292] Step 2:

[0293] The device encrypts the collected audio data and transfers it to the cloud infrastructure using a secure protocol (e.g., HTTPS). The input is the encrypted audio data, and the output is the completion of the transmission to the cloud infrastructure.

[0294] Step 3:

[0295] The server receives audio data via a cloud infrastructure and performs analysis using a machine learning analysis mechanism. The input is the audio data received by the server, and the output is the transcribed conversation content and the results of the fraud probability assessment. For data processing, speech recognition technology (e.g., Google Cloud Speech-to-Text API) is used to convert speech to text, and then a generative AI model performs a fraud risk assessment.

[0296] Step 4:

[0297] Based on the analysis results, the server generates an alert if the fraud risk exceeds a certain threshold. The input is the assessment result of the likelihood of fraud, and the output is the trigger for the warning notification. Specifically, the alarm generation mechanism generates an alert when the risk determination threshold is exceeded.

[0298] Step 5:

[0299] The server utilizes a notification transmission mechanism to send generated alarms to pre-registered communication destinations and the user's mobile device. Inputs are the alarm notification and communication destination information, while output is the transmission of the alarm to recipients. Specifically, the immediate notification mechanism pushes the alarm to the user and simultaneously sends an alert to emergency contacts.

[0300] This series of processes makes it possible to detect fraud risks in real time and respond quickly.

[0301] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0302] This invention aims to improve the accuracy of fraud risk assessment by combining an emotion engine with a system that analyzes user voice data in real time and evaluates the likelihood of fraud. In addition to the conventional functions of collecting and analyzing user voice data, this system further improves fraud risk assessment and alert generation by recognizing the user's emotional state.

[0303] Users first use a smartphone with a dedicated app installed or a device equipped with a dedicated audio data collection system. The device continuously collects audio from the user's surroundings, encrypts it, and temporarily stores it. This audio data is then sent to a cloud server as needed.

[0304] Upon receiving audio data, the server first converts the audio into text using artificial intelligence analysis tools to detect keywords and patterns related to fraud. Next, it uses an emotion engine to extract the user's emotional state from the audio data. This emotional information is considered as one factor in assessing fraud risk.

[0305] For example, if the emotion engine determines that a user is in an excited state, it can prioritize the alert, considering the fraud risk to be higher than usual. In this way, emotional states are used as supplementary information to adjust the fraud risk assessment.

[0306] When the server determines that it is a high risk, it generates an alert by the alert generation means and quickly notifies the user's terminal and the pre-set emergency contacts through the notification means. By receiving the warning from the terminal, the user can take prompt action to prevent fraud damage.

[0307] In this way, by incorporating the emotion engine, it provides a system that improves the detection accuracy of fraud acts and enhances the security for users.

[0308] The following describes the processing flow.

[0309] Step 1:

[0310] The terminal constantly collects the voices around the user with a microphone. This voice data is encrypted for privacy protection and temporarily stored in the terminal.

[0311] Step 2:

[0312] The terminal transfers the collected voice data to the cloud server using a secure protocol while keeping it encrypted. This ensures the confidentiality and tamper prevention of the data.

[0313] Step 3:

[0314] Before analyzing the received voice data, the server converts the voice data into text. Voice recognition technology is used for this conversion process.

[0315] Step 4:

[0316] The server processes the converted text with artificial intelligence analysis means to detect keywords and patterns related to fraud. The detected information is used for the initial assessment of fraud risk.

[0317] Step 5:

[0318] The server uses an emotion engine to extract the user's emotional state from the audio data. Specifically, it analyzes the tone of voice and other factors to determine whether the user is excited, tense, or calm.

[0319] Step 6:

[0320] The server integrates the fraud assessment from step 4 and the emotional state from step 5 to reassess the fraud risk. If the emotional state is associated with a high risk, such as excitement, the risk assessment is adjusted.

[0321] Step 7:

[0322] If the server determines that there is a high risk of fraud, it will generate an alert using an alert generation mechanism. The alert will include the results of the fraud risk assessment and information on the emotional state.

[0323] Step 8:

[0324] The server quickly sends the generated alarm to the user's device and registered emergency contacts. Upon receiving the alarm, the device alerts the user using push notifications and an alert sound.

[0325] Step 9:

[0326] Users receive warnings from their devices and take appropriate action as needed. This allows them to take appropriate measures to prevent becoming a victim of fraud.

[0327] (Example 2)

[0328] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal".

[0329] Fraudulent activities are becoming increasingly sophisticated, making detection extremely difficult. In particular, fraudsters employ psychological tricks to undermine victims' vigilance, limiting the effectiveness of traditional keyword-based detection methods. Furthermore, monitoring systems that fail to adequately consider users' emotional states can lead to inaccurate risk assessments. Therefore, there is a need for more accurate fraud risk assessment systems.

[0330] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0331] In this invention, the server includes acoustic information collection means for continuously acquiring the user's voice information, information transmission means for securely transferring the voice information to an information processing device, and emotion analysis means for extracting emotional states from the voice information and including emotional information in the fraud risk assessment. This makes it possible to assess fraud risk with greater accuracy while taking into account the user's emotional state.

[0332] "User" refers to an individual or group that uses the system.

[0333] "Audio information" refers to the user's speech and surrounding sounds collected as acoustic data.

[0334] "Acoustic information acquisition means" refers to a device or method used to acquire the user's voice information.

[0335] An "information processing device" refers to a computer system used for data analysis, storage, and transfer.

[0336] "Information transmission means" refers to technical means for securely transferring collected audio information to a server or cloud.

[0337] "Machine learning analysis methods" refer to AI technologies and algorithms used to analyze voice information and assess the likelihood of fraud.

[0338] "Emotional analysis means" refers to a technology or algorithm for extracting the user's emotional state from voice information.

[0339] "Alert generation means" refers to a function that generates warnings or alerts when the risk of fraud exceeds a certain threshold.

[0340] "Notification means" refers to a communication means used to transmit generated alarms to users or pre-registered communication partners.

[0341] This invention combines a sentiment analysis function with a fraud risk assessment system that analyzes voice data in real time to enhance user safety. Specific embodiments are described below.

[0342] First, the user uses a mobile device or acoustic information collection device with a dedicated application installed that can collect voice information. The device uses a built-in microphone to continuously acquire the user's voice information and temporarily stores it as digital data. The data is protected from unauthorized access by others using encryption methods such as AES.

[0343] After the terminal collects voice information, it securely transmits the encrypted voice information to an information processing device in the cloud using an information transmission method. Communication is conducted via the HTTPS protocol, guaranteeing data integrity and confidentiality.

[0344] The server analyzes the received data. First, it uses machine learning analysis to convert the audio data into text data. For general text conversion, natural language processing models, known as speech recognition technology, are used. Furthermore, the server uses sentiment analysis to extract the user's emotional state from the audio data. Sentiment analysis employs algorithms that analyze the tone and rhythm of the voice.

[0345] If the server determines that there is a high risk of fraud, it will notify the user via an alarm generation system. This includes push notifications to the device and email notifications to registered communication partners. This allows users to take immediate action.

[0346] For example, when a user suddenly starts using a particular word frequently, emotional analysis can detect tension or excitement. As a result, the risk of fraud is assessed as higher than usual. An example of a prompt message that might be input to the generating AI model is, "Analyze the emotional signs related to fraud from this audio data."

[0347] By incorporating sentiment analysis in this way, we can provide a system that offers more accurate fraud risk assessments and supports users' safer lives.

[0348] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0349] Step 1:

[0350] The terminal acquires audio information captured by the acoustic information collection device. The microphone detects all sounds the user makes and captures them as digital audio data. At this point, the input is an analog audio signal, and the output is digital audio data. The digitized audio information is temporarily stored on the terminal. During this time, the data is protected by AES encryption.

[0351] Step 2:

[0352] The terminal uses an information transmission method to send encrypted voice information to a server in the cloud. The input here is encrypted digital voice data, and the output is data securely transferred to the server. The HTTPS protocol ensures data integrity and confidentiality.

[0353] Step 3:

[0354] The server uses machine learning analysis to convert audio information into text. It receives digital audio data as input and outputs text data. By utilizing an acoustic model to convert audio into text, it creates a format suitable for analysis.

[0355] Step 4:

[0356] The server uses emotion analysis tools to analyze transcribed data and voice tone to extract the user's emotional state. The inputs are text data and digital voice data, while the output is data indicating the emotional state. The emotion analysis model detects excitement and tension based on the intonation and speed of the voice.

[0357] Step 5:

[0358] The server assesses the risk of fraud based on the analyzed data. The input consists of keyword detection results and sentiment information, while the output is a fraud risk assessment index. This assessment uses a generative AI model to determine the risk according to the prompt "Assess the fraud risk from this audio data."

[0359] Step 6:

[0360] The server issues an alert using an alarm generation mechanism if it determines that the fraud risk is high. The input is a fraud risk assessment index, and the output is the generated alarm. The alarm is pushed to the device and, in some cases, also notified to emergency contacts.

[0361] This series of processes allows users to respond quickly to fraud, improving security.

[0362] (Application Example 2)

[0363] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the smart glasses 214 will be referred to as the "terminal."

[0364] In modern society, while online payments and voice-based transactions are increasing, fraudulent activities are also becoming more sophisticated. Therefore, there is a growing need for systems that can analyze voice data in real time and assess fraud risk with greater accuracy. In particular, there is a demand for systems that can interpret emotions from voice and assess fraud risk while taking the user's psychological state into account, but achieving this presents significant challenges.

[0365] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0366] In this invention, the server includes acoustic data collection means for continuously acquiring user voice information, data transmission means for encrypting the voice information and transferring it to a remote computer, artificial intelligence analysis means for converting the voice information into text and detecting words and patterns related to fraud, and an emotion analysis engine for recognizing emotional states and reflecting them in the fraud possibility assessment. This makes it possible to accurately assess the risk of fraud using emotion information extracted from voice data and appropriately notify the user of a warning.

[0367] "Acoustic data acquisition means" refers to devices or programs that continuously acquire ambient sound information, and have the function of collecting sound via smartphones or microphones.

[0368] "Data transmission means" refers to technologies and methods for encrypting collected audio information and securely transferring it to a remote computer.

[0369] "Artificial intelligence analysis means" refers to technologies and algorithms used to convert audio information into text and detect words and patterns related to fraud from that text.

[0370] A "sentiment analysis engine" refers to analytical technology that recognizes a user's emotional state based on voice information and reflects it in relation to the assessment of the risk of fraud.

[0371] An "alert generation method" refers to a device or method that generates an alert and notifies the user or registered contacts when the risk level of fraud exceeds a predetermined threshold.

[0372] "Notification method" refers to the technology or method used to send generated alarms to users or pre-registered contacts.

[0373] This invention is realized by equipping the user's terminal with acoustic data collection means. The terminal constantly collects audio from the user's surroundings, encrypts it, and temporarily stores it. This encrypted audio information is transferred to a cloud server in a secure manner using data transmission means as needed.

[0374] The server converts the received audio information into text using artificial intelligence analysis tools to detect words and patterns related to fraud. Software such as Google Cloud Speech-to-Text and AWS Transcribe can be used in this process. Furthermore, a sentiment analysis engine is utilized to identify the user's emotional state from the audio information. IBM Watson Tone Analyzer and Microsoft Text Analytics are suitable for this analysis.

[0375] The server combines data obtained from emotional states and voice information to assess the risk of fraud. If the risk of fraud exceeds a threshold, an alert generation system creates a warning and quickly sends the warning to the user's device or pre-registered contacts via a notification system.

[0376] For example, if an analysis of the audio generated when a user attempts to purchase an electronic device online detects an abnormal level of excitement, it is determined that the user is likely to be a fraud. In this situation, the user is warned to perform additional verification, such as by being prompted to enable two-factor authentication.

[0377] An example of a prompt is: "Detect whether the user is experiencing emotional distress during online payment and assess the likelihood of fraud."

[0378] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0379] Step 1:

[0380] The device continuously collects audio from the user's surroundings using acoustic data collection equipment. This audio data is acquired in real time as input and temporarily stored within the device. The data is encrypted and prepared for transmission to a cloud server in a secure state.

[0381] Step 2:

[0382] The terminal transfers encrypted voice information to a cloud server using a data transmission method. In this process, data is transferred over the network and prepared for reception by the server. The input is encrypted voice data, and the server receives the data as output.

[0383] Step 3:

[0384] The server uses artificial intelligence analysis to convert the received audio data into text. Here, Google Cloud Speech-to-Text is used as the data processing tool to convert audio data to text. The input to this process is audio data, and the output is text data in which fraud-related terms can be detected.

[0385] Step 4:

[0386] The server uses a sentiment analysis engine to determine the user's emotional state from text data. The input is the text data generated in step 3, and the output is information about the user's emotional state. IBM Watson Tone Analyzer is used for this sentiment assessment.

[0387] Step 5:

[0388] The server assesses the risk of fraud based on fraud-related terms and sentiment detected by artificial intelligence analysis. The input is text and sentiment data related to fraud risk, and the output is a risk assessment. If this risk level exceeds a threshold, an alert generation mechanism is triggered.

[0389] Step 6:

[0390] An alert is generated by the alert generation mechanism. This alert suggests a high probability of fraud and is ready to be sent as output to the user's device and registered contacts.

[0391] Step 7:

[0392] The server uses a notification system to send generated alarms to the user's device and designated contacts. The input is the generated alarm, and the output is a notification sent to the user and contacts. This allows the user to take immediate action.

[0393] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0394] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0395] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the smart glasses 214.

[0396] [Third Embodiment]

[0397] Figure 5 shows an example of the configuration of the data processing system 310 according to the third embodiment.

[0398] As shown in Figure 5, the data processing system 310 includes a data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0399] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0400] The headset terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a display 343. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and display 343 are also connected to the bus 52.

[0401] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0402] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0403] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0404] Figure 6 shows an example of the main functions of the data processing device 12 and the headset terminal 314. As shown in Figure 6, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0405] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0406] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0407] In the headset terminal 314, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0408] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the headset terminal 314 will be referred to as the "terminal".

[0409] This invention is implemented as a voice analysis system to prevent deposit fraud targeting the elderly. This system continuously acquires the user's voice, analyzes it in real time, and functions to detect signs of fraud early and issue a warning.

[0410] Users first need either a device equipped with acoustic data collection capabilities or a smartphone with a dedicated application installed. This device has the function of collecting ambient sounds using a microphone, encrypting the data within the device, and temporarily storing it. The collected audio data is then transferred to a cloud server using a secure protocol.

[0411] The server processes the received audio data using its built-in artificial intelligence analysis capabilities to accurately assess the likelihood of fraud. The analysis involves transcribing the audio into text and comparing it with a database of past fraud cases to identify keywords and conversational patterns specific to fraud. When signs of fraud are detected, the server generates a warning using an alert generation mechanism and quickly sends this information to the user's device.

[0412] The device receives the warning and alerts the user with a push notification and an alert sound. Simultaneously, a notification is sent to pre-set emergency contacts to encourage a quick response. For example, if a user receives a suspicious investment solicitation over the phone, the audio is analyzed in real time, and if it is determined to be potentially fraudulent, an automatic notification is sent to family members or the police immediately.

[0413] In this way, this system provides users with peace of mind by constantly monitoring the risk of fraud lurking in everyday communication and responding quickly.

[0414] The following describes the processing flow.

[0415] Step 1:

[0416] The device constantly collects sounds from the user's surroundings using a microphone and captures them as audio data. This audio data is encrypted for privacy protection and temporarily stored on the device.

[0417] Step 2:

[0418] The terminal transfers pre-processed audio data to the cloud server using a secure protocol. During this process, the data remains encrypted and protected from tampering.

[0419] Step 3:

[0420] The server receives the audio data sent to the cloud and analyzes the audio using artificial intelligence analysis tools. This analysis converts the audio into text and detects keywords and conversation patterns specific to fraud.

[0421] Step 4:

[0422] The server compares detected keywords and patterns with a database of past fraud cases to assess the risk of fraud. Based on this assessment, it determines whether the case is highly likely to be fraudulent.

[0423] Step 5:

[0424] The server generates an alert using an alert generation mechanism if it determines that the risk of fraud exceeds a certain threshold. This alert includes the fraud assessment results and a text sample of the audio.

[0425] Step 6:

[0426] The server sends the generated alarm to the user's device and pre-registered emergency contacts. Upon receiving the alarm, the user's device is notified via push notification or an alert sound.

[0427] Step 7:

[0428] Users receive warnings from their devices and are prompted to take appropriate action. This allows users and stakeholders to take swift action to prevent fraud.

[0429] (Example 1)

[0430] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0431] Fraudulent activities targeting the elderly are becoming more sophisticated year by year, and there is a need for effective means to prevent them from becoming victims. However, conventional methods make it difficult to detect fraud in real time and provide rapid notification, and there are also issues of user privacy. The present invention aims to solve these problems and provide a system that can detect the possibility of fraud early with high accuracy and quickly notify users and relevant parties of that information.

[0432] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0433] In this invention, the server includes machine learning analysis means for analyzing voice information, means for converting voice to text using natural language processing technology, and warning generation means for generating alarms and providing notifications. This enables real-time analysis of voice information, allowing for highly accurate detection of signs of fraud and the rapid and secure transmission of warnings to relevant parties.

[0434] "Audio information acquisition means" refers to a device or technology that continuously collects sounds from the user's surroundings.

[0435] "Information transmission means" refers to protocols and devices for securely transferring acquired voice information over a communication network.

[0436] "Machine learning analysis methods" refer to techniques that use artificial intelligence technology to analyze voice information and assess the likelihood of fraud.

[0437] A "warning generation method" is a technology that automatically generates an alarm when the risk of fraud exceeds a certain threshold.

[0438] A "notification means" refers to a method or device for transmitting a generated alarm to a pre-registered communication destination.

[0439] "Natural language processing technology" is a technology for converting spoken information into text and analyzing its content.

[0440] An "information display means" is an interface for visually or audibly communicating warnings or notifications to the user.

[0441] This invention relates to a voice analysis system for protecting the elderly from fraud. The system achieves its specific functions through the cooperation of a user, a terminal, and a server. The user uses a mobile terminal with a dedicated application installed to collect voice information. This terminal is equipped with hardware and software to acquire ambient sounds with a microphone, encrypt the data, and temporarily store it.

[0442] The device transmits the acquired voice information to a server in the cloud using a secure protocol. HTTPS is often used as the protocol. The server converts the voice to text and compares it with a database containing past fraud cases. This process utilizes speech recognition and natural language processing technologies.

[0443] The server incorporates a generative AI model to analyze voice information and detect fraudulent keywords and conversation patterns. If it determines that there is a high probability of fraud, the server immediately generates an alarm and notifies the device of this information. The notification is conveyed to the user through the device's push notification function or visual alert.

[0444] For example, when a user receives a suspicious phone solicitation, the system analyzes the conversation in real time and, if it determines the content to be fraudulent, immediately generates an alert and notifies family members and the police. This protects users from fraud in real time.

[0445] Furthermore, as an example of prompts to the generating AI model, questions such as "How can AI be used to issue real-time warnings if an elderly person is about to become a victim of bank deposit fraud?" can be asked. This system will protect users from the risk of fraud and enable them to live a safer life.

[0446] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0447] Step 1:

[0448] The user activates a device equipped with an acoustic information acquisition mechanism and begins collecting audio data. The device uses a microphone to acquire ambient acoustic signals and encrypts this data within the device. The input is the ambient audio signals received by the microphone, and the output is the encrypted audio data. Specifically, when the user launches a dedicated application, the device automatically begins collecting audio.

[0449] Step 2:

[0450] The device transfers encrypted audio data to a server in the cloud using a secure protocol. The input is encrypted audio data stored on the device, and the output is encrypted data sent to the server. Specifically, data transmission uses the HTTPS protocol and is performed in the background without user intervention.

[0451] Step 3:

[0452] The server decrypts the received encrypted audio data and converts it to text using natural language processing techniques. The input is the encrypted audio data that arrives at the server, and the output is the audio information converted into text. Specifically, the server runs a speech recognition algorithm to accurately convert the audio into text.

[0453] Step 4:

[0454] The server analyzes transcribed audio data using machine learning analysis tools to assess the risk of fraud. The input is transcribed audio information, and the output is the fraud risk assessment result. Specifically, the server analyzes keywords and conversation patterns specific to fraud and scores the likelihood of fraud using a generative AI model.

[0455] Step 5:

[0456] If the server determines that there is a high probability of fraud, it immediately generates an alarm and notifies the terminal of this information. The input is the result of the fraud risk assessment, and the output is the generated alarm message. Specifically, the server creates the warning message in real time and sends it to the terminal via the notification system.

[0457] Step 6:

[0458] The device notifies the user of received alarms via push notifications and alert sounds. The input is the alarm message sent from the server, and the output is the warning notification to the user. Specifically, the device displays the warning message on the screen and simultaneously sounds an alert to draw the user's attention.

[0459] This allows the system to detect signs of fraud with high accuracy in real time, supporting a swift response.

[0460] (Application Example 1)

[0461] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0462] To address the vulnerability of the elderly to fraudulent verbal transactions such as bank deposit scams, technology is needed that analyzes voices in real time, quickly detects signs of fraud, and implements appropriate protective measures. Furthermore, there is a need for a system that is simple and automated for the elderly, providing a sense of security and safety.

[0463] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0464] In this invention, the server includes a voice acquisition mechanism for continuously acquiring user voice information, an information transmission mechanism for securely transferring voice information to a cloud infrastructure, and a machine learning analysis mechanism for evaluating the likelihood of fraud. This makes it possible to quickly detect signs of fraud, generate an alarm, and immediately notify the user and registered emergency contacts.

[0465] A "voice acquisition mechanism" is a device that plays the role of continuously collecting the user's voice information.

[0466] An "information transmission mechanism" is a device that transfers collected voice information to a cloud infrastructure via a secure communication protocol.

[0467] A "machine learning analysis mechanism" is a device that uses artificial intelligence technology to analyze collected audio information and evaluate the likelihood of fraud.

[0468] An "alarm generation mechanism" is a device that generates an alarm when the risk of fraud exceeds a certain threshold.

[0469] A "notification transmission mechanism" is a device that transmits generated alarms to pre-registered communication destinations.

[0470] An "instant notification mechanism" is a device that instantly notifies the user's mobile device or communication partner of an alarm.

[0471] An "information aggregation platform" is a database system that stores data on past fraud cases to help determine the likelihood of fraud.

[0472] The system realizing this invention begins with a terminal equipped with a voice acquisition mechanism that the user wears or carries. The terminal continuously collects ambient sound using a highly sensitive microphone and securely transmits this voice information to a cloud platform. In the cloud platform, the voice information is analyzed by a machine learning analysis mechanism to assess the likelihood of fraud. The analysis uses speech recognition technologies such as the Google Cloud Speech-to-Text API and a generative AI model that has learned the characteristics of fraud.

[0473] Based on the analysis results, the server generates an alert if the risk of fraud exceeds a certain threshold. When an alert is generated, it is sent to pre-registered contacts via a notification system. In addition, an instant notification system immediately notifies the user's mobile device, promptly alerting the user and their family.

[0474] For example, if a user receives a suspicious investment solicitation over the phone, the system collects the audio recording. If the recording is identified as containing dangerous keywords such as "Please transfer the money immediately," the system quickly generates an alert to inform the user of the situation. An alert is also immediately sent to pre-registered emergency contacts, such as family members' smartphones, allowing them to take immediate action to protect the user.

[0475] An example of a prompt for a generative AI model would be: "Analyze this voice conversation to see if it contains signs of fraud. The text is: 'Please provide your account number now...'"

[0476] This system makes it possible to prevent the risk of fraud targeting the elderly and provide a safe and secure living environment.

[0477] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0478] Step 1:

[0479] When the device is powered on, it uses its voice acquisition mechanism to collect ambient sounds. The input is ambient sound, and the output is audio data converted into a digital format. This data is temporarily stored within the device.

[0480] Step 2:

[0481] The device encrypts the collected audio data and transfers it to the cloud infrastructure using a secure protocol (e.g., HTTPS). The input is the encrypted audio data, and the output is the completion of the transmission to the cloud infrastructure.

[0482] Step 3:

[0483] The server receives audio data via a cloud infrastructure and performs analysis using a machine learning analysis mechanism. The input is the audio data received by the server, and the output is the transcribed conversation content and the results of the fraud probability assessment. For data processing, speech recognition technology (e.g., Google Cloud Speech-to-Text API) is used to convert speech to text, and then a generative AI model performs a fraud risk assessment.

[0484] Step 4:

[0485] Based on the analysis results, the server generates an alert if the fraud risk exceeds a certain threshold. The input is the assessment result of the likelihood of fraud, and the output is the trigger for the warning notification. Specifically, the alarm generation mechanism generates an alert when the risk determination threshold is exceeded.

[0486] Step 5:

[0487] The server utilizes a notification transmission mechanism to send generated alarms to pre-registered communication destinations and the user's mobile device. Inputs are the alarm notification and communication destination information, while output is the transmission of the alarm to recipients. Specifically, the immediate notification mechanism pushes the alarm to the user and simultaneously sends an alert to emergency contacts.

[0488] This series of processes makes it possible to detect fraud risks in real time and respond quickly.

[0489] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0490] This invention aims to improve the accuracy of fraud risk assessment by combining an emotion engine with a system that analyzes user voice data in real time and evaluates the likelihood of fraud. In addition to the conventional functions of collecting and analyzing user voice data, this system further improves fraud risk assessment and alert generation by recognizing the user's emotional state.

[0491] Users first use a smartphone with a dedicated app installed or a device equipped with a dedicated audio data collection system. The device continuously collects audio from the user's surroundings, encrypts it, and temporarily stores it. This audio data is then sent to a cloud server as needed.

[0492] Upon receiving audio data, the server first converts the audio into text using artificial intelligence analysis tools to detect keywords and patterns related to fraud. Next, it uses an emotion engine to extract the user's emotional state from the audio data. This emotional information is considered as one factor in assessing fraud risk.

[0493] For example, if the emotion engine determines that a user is in an excited state, it can prioritize the alert, considering the fraud risk to be higher than usual. In this way, emotional states are used as supplementary information to adjust the fraud risk assessment.

[0494] If the server determines that the situation is high-risk, it generates an alert using an alert generation mechanism and promptly notifies the user's device and pre-configured emergency contacts via a notification mechanism. By receiving the warning on their device, the user can take swift action to prevent fraud.

[0495] In this way, by incorporating an emotion engine, we can improve the accuracy of detecting fraudulent activity and provide a system that enhances user safety.

[0496] The following describes the processing flow.

[0497] Step 1:

[0498] The device constantly collects audio from the user's surroundings using its microphone. This audio data is encrypted for privacy protection and temporarily stored on the device.

[0499] Step 2:

[0500] The device encrypts the collected audio data and transfers it to the cloud server using a secure protocol. This ensures data confidentiality and prevents tampering.

[0501] Step 3:

[0502] Before analyzing the received audio data, the server converts it into text. This conversion process uses speech recognition technology.

[0503] Step 4:

[0504] The server processes the converted text using artificial intelligence analysis to detect keywords and patterns related to fraud. The detected information is used for an initial assessment of fraud risk.

[0505] Step 5:

[0506] The server uses an emotion engine to extract the user's emotional state from the audio data. Specifically, it analyzes the tone of voice and other factors to determine whether the user is excited, tense, or calm.

[0507] Step 6:

[0508] The server integrates the fraud assessment from step 4 and the emotional state from step 5 to reassess the fraud risk. If the emotional state is associated with a high risk, such as excitement, the risk assessment is adjusted.

[0509] Step 7:

[0510] If the server determines that there is a high risk of fraud, it will generate an alert using an alert generation mechanism. The alert will include the results of the fraud risk assessment and information on the emotional state.

[0511] Step 8:

[0512] The server quickly sends the generated alarm to the user's device and registered emergency contacts. Upon receiving the alarm, the device alerts the user using push notifications and an alert sound.

[0513] Step 9:

[0514] Users receive warnings from their devices and take appropriate action as needed. This allows them to take appropriate measures to prevent becoming a victim of fraud.

[0515] (Example 2)

[0516] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0517] Fraudulent activities are becoming increasingly sophisticated, making detection extremely difficult. In particular, fraudsters employ psychological tricks to undermine victims' vigilance, limiting the effectiveness of traditional keyword-based detection methods. Furthermore, monitoring systems that fail to adequately consider users' emotional states can lead to inaccurate risk assessments. Therefore, there is a need for more accurate fraud risk assessment systems.

[0518] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0519] In this invention, the server includes acoustic information collection means for continuously acquiring the user's voice information, information transmission means for securely transferring the voice information to an information processing device, and emotion analysis means for extracting emotional states from the voice information and including emotional information in the fraud risk assessment. This makes it possible to assess fraud risk with greater accuracy while taking into account the user's emotional state.

[0520] "User" refers to an individual or group that uses the system.

[0521] "Audio information" refers to the user's speech and surrounding sounds collected as acoustic data.

[0522] "Acoustic information acquisition means" refers to a device or method used to acquire the user's voice information.

[0523] An "information processing device" refers to a computer system used for data analysis, storage, and transfer.

[0524] "Information transmission means" refers to technical means for securely transferring collected audio information to a server or cloud.

[0525] "Machine learning analysis methods" refer to AI technologies and algorithms used to analyze voice information and assess the likelihood of fraud.

[0526] "Emotional analysis means" refers to a technology or algorithm for extracting the user's emotional state from voice information.

[0527] "Alert generation means" refers to a function that generates warnings or alerts when the risk of fraud exceeds a certain threshold.

[0528] "Notification means" refers to a communication means used to transmit generated alarms to users or pre-registered communication partners.

[0529] This invention combines a sentiment analysis function with a fraud risk assessment system that analyzes voice data in real time to enhance user safety. Specific embodiments are described below.

[0530] First, the user uses a mobile device or acoustic information collection device with a dedicated application installed that can collect voice information. The device uses a built-in microphone to continuously acquire the user's voice information and temporarily stores it as digital data. The data is protected from unauthorized access by others using encryption methods such as AES.

[0531] After the terminal collects voice information, it securely transmits the encrypted voice information to an information processing device in the cloud using an information transmission method. Communication is conducted via the HTTPS protocol, guaranteeing data integrity and confidentiality.

[0532] The server analyzes the received data. First, it uses machine learning analysis to convert the audio data into text data. For general text conversion, natural language processing models, known as speech recognition technology, are used. Furthermore, the server uses sentiment analysis to extract the user's emotional state from the audio data. Sentiment analysis employs algorithms that analyze the tone and rhythm of the voice.

[0533] If the server determines that there is a high risk of fraud, it will notify the user via an alarm generation system. This includes push notifications to the device and email notifications to registered communication partners. This allows users to take immediate action.

[0534] For example, when a user suddenly starts using a particular word frequently, emotional analysis can detect tension or excitement. As a result, the risk of fraud is assessed as higher than usual. An example of a prompt message that might be input to the generating AI model is, "Analyze the emotional signs related to fraud from this audio data."

[0535] By incorporating sentiment analysis in this way, we can provide a system that offers more accurate fraud risk assessments and supports users' safer lives.

[0536] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0537] Step 1:

[0538] The terminal acquires audio information captured by the acoustic information collection device. The microphone detects all sounds the user makes and captures them as digital audio data. At this point, the input is an analog audio signal, and the output is digital audio data. The digitized audio information is temporarily stored on the terminal. During this time, the data is protected by AES encryption.

[0539] Step 2:

[0540] The terminal uses an information transmission method to send encrypted voice information to a server in the cloud. The input here is encrypted digital voice data, and the output is data securely transferred to the server. The HTTPS protocol ensures data integrity and confidentiality.

[0541] Step 3:

[0542] The server uses machine learning analysis to convert audio information into text. It receives digital audio data as input and outputs text data. By utilizing an acoustic model to convert audio into text, it creates a format suitable for analysis.

[0543] Step 4:

[0544] The server uses emotion analysis tools to analyze transcribed data and voice tone to extract the user's emotional state. The inputs are text data and digital voice data, while the output is data indicating the emotional state. The emotion analysis model detects excitement and tension based on the intonation and speed of the voice.

[0545] Step 5:

[0546] The server assesses the risk of fraud based on the analyzed data. The input consists of keyword detection results and sentiment information, while the output is a fraud risk assessment index. This assessment uses a generative AI model to determine the risk according to the prompt "Assess the fraud risk from this audio data."

[0547] Step 6:

[0548] The server issues an alert using an alarm generation mechanism if it determines that the fraud risk is high. The input is a fraud risk assessment index, and the output is the generated alarm. The alarm is pushed to the device and, in some cases, also notified to emergency contacts.

[0549] This series of processes allows users to respond quickly to fraud, improving security.

[0550] (Application Example 2)

[0551] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server," and the headset-type terminal 314 will be referred to as the "terminal."

[0552] In modern society, while online payments and voice-based transactions are increasing, fraudulent activities are also becoming more sophisticated. Therefore, there is a growing need for systems that can analyze voice data in real time and assess fraud risk with greater accuracy. In particular, there is a demand for systems that can interpret emotions from voice and assess fraud risk while taking the user's psychological state into account, but achieving this presents significant challenges.

[0553] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0554] In this invention, the server includes acoustic data collection means for continuously acquiring user voice information, data transmission means for encrypting the voice information and transferring it to a remote computer, artificial intelligence analysis means for converting the voice information into text and detecting words and patterns related to fraud, and an emotion analysis engine for recognizing emotional states and reflecting them in the fraud possibility assessment. This makes it possible to accurately assess the risk of fraud using emotion information extracted from voice data and appropriately notify the user of a warning.

[0555] "Acoustic data acquisition means" refers to devices or programs that continuously acquire ambient sound information, and have the function of collecting sound via smartphones or microphones.

[0556] "Data transmission means" refers to technologies and methods for encrypting collected audio information and securely transferring it to a remote computer.

[0557] "Artificial intelligence analysis means" refers to technologies and algorithms used to convert audio information into text and detect words and patterns related to fraud from that text.

[0558] A "sentiment analysis engine" refers to analytical technology that recognizes a user's emotional state based on voice information and reflects it in relation to the assessment of the risk of fraud.

[0559] An "alert generation method" refers to a device or method that generates an alert and notifies the user or registered contacts when the risk level of fraud exceeds a predetermined threshold.

[0560] "Notification method" refers to the technology or method used to send generated alarms to users or pre-registered contacts.

[0561] This invention is realized by equipping the user's terminal with acoustic data collection means. The terminal constantly collects audio from the user's surroundings, encrypts it, and temporarily stores it. This encrypted audio information is transferred to a cloud server in a secure manner using data transmission means as needed.

[0562] The server converts the received audio information into text using artificial intelligence analysis tools to detect words and patterns related to fraud. Software such as Google Cloud Speech-to-Text and AWS Transcribe can be used in this process. Furthermore, a sentiment analysis engine is utilized to identify the user's emotional state from the audio information. IBM Watson Tone Analyzer and Microsoft Text Analytics are suitable for this analysis.

[0563] The server combines data obtained from emotional states and voice information to assess the risk of fraud. If the risk of fraud exceeds a threshold, an alert generation system creates a warning and quickly sends the warning to the user's device or pre-registered contacts via a notification system.

[0564] For example, if an analysis of the audio generated when a user attempts to purchase an electronic device online detects an abnormal level of excitement, it is determined that the user is likely to be a fraud. In this situation, the user is warned to perform additional verification, such as by being prompted to enable two-factor authentication.

[0565] An example of a prompt is: "Detect whether the user is experiencing emotional distress during online payment and assess the likelihood of fraud."

[0566] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0567] Step 1:

[0568] The device continuously collects audio from the user's surroundings using acoustic data collection equipment. This audio data is acquired in real time as input and temporarily stored within the device. The data is encrypted and prepared for transmission to a cloud server in a secure state.

[0569] Step 2:

[0570] The terminal transfers encrypted voice information to a cloud server using a data transmission method. In this process, data is transferred over the network and prepared for reception by the server. The input is encrypted voice data, and the server receives the data as output.

[0571] Step 3:

[0572] The server uses artificial intelligence analysis to convert the received audio data into text. Here, Google Cloud Speech-to-Text is used as the data processing tool to convert audio data to text. The input to this process is audio data, and the output is text data in which fraud-related terms can be detected.

[0573] Step 4:

[0574] The server uses a sentiment analysis engine to determine the user's emotional state from text data. The input is the text data generated in step 3, and the output is information about the user's emotional state. IBM Watson Tone Analyzer is used for this sentiment assessment.

[0575] Step 5:

[0576] The server assesses the risk of fraud based on fraud-related terms and sentiment detected by artificial intelligence analysis. The input is text and sentiment data related to fraud risk, and the output is a risk assessment. If this risk level exceeds a threshold, an alert generation mechanism is triggered.

[0577] Step 6:

[0578] An alert is generated by the alert generation mechanism. This alert suggests a high probability of fraud and is ready to be sent as output to the user's device and registered contacts.

[0579] Step 7:

[0580] The server uses a notification system to send generated alarms to the user's device and designated contacts. The input is the generated alarm, and the output is a notification sent to the user and contacts. This allows the user to take immediate action.

[0581] The specific processing unit 290 transmits the result of the specific processing to the headset terminal 314. In the headset terminal 314, the control unit 46A causes the speaker 240 and display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0582] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0583] In the above embodiment, an example was given in which specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and specific processing may also be performed by the headset terminal 314.

[0584] [Fourth Embodiment]

[0585] Figure 7 shows an example of the configuration of the data processing system 410 according to the fourth embodiment.

[0586] As shown in Figure 7, the data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0587] The data processing device 12 comprises a computer 22, a database 24, and a communication interface 26. The computer 22 is an example of a "computer" related to the technology of this disclosure. The computer 22 comprises a processor 28, RAM 30, and storage 32. The processor 28, RAM 30, and storage 32 are connected to a bus 34. The database 24 and the communication interface 26 are also connected to the bus 34. The communication interface 26 is connected to a network 54. An example of the network 54 is a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0588] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication interface 44, and a controlled object 443. The computer 36 includes a processor 46, RAM 48, and storage 50. The processor 46, RAM 48, and storage 50 are connected to a bus 52. The microphone 238, speaker 240, camera 42, and controlled object 443 are also connected to the bus 52.

[0589] The microphone 238 receives voice signals from the user 20 and receives instructions from the user 20. The microphone 238 captures the voice signals from the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio according to the instructions from the processor 46.

[0590] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an image sensor such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the area around the user 20 (for example, an imaging range defined by a field of view equivalent to the width of a typical healthy person's field of vision).

[0591] Communication interface 44 is connected to network 54. Communication interfaces 44 and 26 are responsible for the exchange of various information between processor 46 and processor 28 via network 54. The exchange of various information between processor 46 and processor 28 using communication interfaces 44 and 26 is performed in a secure manner.

[0592] The controlled object 443 includes a display device, LEDs in the eyes, and motors that drive the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the robot 414's emotions can be expressed by controlling these motors. Furthermore, the robot 414's facial expressions can also be expressed by controlling the illumination state of the LEDs in its eyes.

[0593] Figure 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Figure 8, the data processing device 12 performs specific processing using the processor 28. The storage 32 stores the specific processing program 56.

[0594] The specific processing program 56 is an example of a "program" relating to the technology of this disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0595] The storage 32 stores the data generation model 58 and the emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0596] In robot 414, the processor 46 performs the reception output processing. The storage 50 stores the reception output program 60. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as a control unit 46A according to the reception output program 60 executed on the RAM 48.

[0597] Next, the specific processing performed by the specific processing unit 290 of the data processing device 12 will be described. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0598] This invention is implemented as a voice analysis system to prevent deposit fraud targeting the elderly. This system continuously acquires the user's voice, analyzes it in real time, and functions to detect signs of fraud early and issue a warning.

[0599] Users first need either a device equipped with acoustic data collection capabilities or a smartphone with a dedicated application installed. This device has the function of collecting ambient sounds using a microphone, encrypting the data within the device, and temporarily storing it. The collected audio data is then transferred to a cloud server using a secure protocol.

[0600] The server processes the received audio data using its built-in artificial intelligence analysis capabilities to accurately assess the likelihood of fraud. The analysis involves transcribing the audio into text and comparing it with a database of past fraud cases to identify keywords and conversational patterns specific to fraud. When signs of fraud are detected, the server generates a warning using an alert generation mechanism and quickly sends this information to the user's device.

[0601] The device receives the warning and alerts the user with a push notification and an alert sound. Simultaneously, a notification is sent to pre-set emergency contacts to encourage a quick response. For example, if a user receives a suspicious investment solicitation over the phone, the audio is analyzed in real time, and if it is determined to be potentially fraudulent, an automatic notification is sent to family members or the police immediately.

[0602] In this way, this system provides users with peace of mind by constantly monitoring the risk of fraud lurking in everyday communication and responding quickly.

[0603] The following describes the processing flow.

[0604] Step 1:

[0605] The device constantly collects sounds from the user's surroundings using a microphone and captures them as audio data. This audio data is encrypted for privacy protection and temporarily stored on the device.

[0606] Step 2:

[0607] The terminal transfers pre-processed audio data to the cloud server using a secure protocol. During this process, the data remains encrypted and protected from tampering.

[0608] Step 3:

[0609] The server receives the audio data sent to the cloud and analyzes the audio using artificial intelligence analysis tools. This analysis converts the audio into text and detects keywords and conversation patterns specific to fraud.

[0610] Step 4:

[0611] The server compares detected keywords and patterns with a database of past fraud cases to assess the risk of fraud. Based on this assessment, it determines whether the case is highly likely to be fraudulent.

[0612] Step 5:

[0613] The server generates an alert using an alert generation mechanism if it determines that the risk of fraud exceeds a certain threshold. This alert includes the fraud assessment results and a text sample of the audio.

[0614] Step 6:

[0615] The server sends the generated alarm to the user's device and pre-registered emergency contacts. Upon receiving the alarm, the user's device is notified via push notification or an alert sound.

[0616] Step 7:

[0617] Users receive warnings from their devices and are prompted to take appropriate action. This allows users and stakeholders to take swift action to prevent fraud.

[0618] (Example 1)

[0619] Next, we will describe Example 1. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0620] Fraudulent activities targeting the elderly are becoming more sophisticated year by year, and there is a need for effective means to prevent them from becoming victims. However, conventional methods make it difficult to detect fraud in real time and provide rapid notification, and there are also issues of user privacy. The present invention aims to solve these problems and provide a system that can detect the possibility of fraud early with high accuracy and quickly notify users and relevant parties of that information.

[0621] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 1 is realized by the following means.

[0622] In this invention, the server includes machine learning analysis means for analyzing voice information, means for converting voice to text using natural language processing technology, and warning generation means for generating alarms and providing notifications. This enables real-time analysis of voice information, allowing for highly accurate detection of signs of fraud and the rapid and secure transmission of warnings to relevant parties.

[0623] "Audio information acquisition means" refers to a device or technology that continuously collects sounds from the user's surroundings.

[0624] "Information transmission means" refers to protocols and devices for securely transferring acquired voice information over a communication network.

[0625] "Machine learning analysis methods" refer to techniques that use artificial intelligence technology to analyze voice information and assess the likelihood of fraud.

[0626] A "warning generation method" is a technology that automatically generates an alarm when the risk of fraud exceeds a certain threshold.

[0627] A "notification means" refers to a method or device for transmitting a generated alarm to a pre-registered communication destination.

[0628] "Natural language processing technology" is a technology for converting spoken information into text and analyzing its content.

[0629] An "information display means" is an interface for visually or audibly communicating warnings or notifications to the user.

[0630] This invention relates to a voice analysis system for protecting the elderly from fraud. The system achieves its specific functions through the cooperation of a user, a terminal, and a server. The user uses a mobile terminal with a dedicated application installed to collect voice information. This terminal is equipped with hardware and software to acquire ambient sounds with a microphone, encrypt the data, and temporarily store it.

[0631] The device transmits the acquired voice information to a server in the cloud using a secure protocol. HTTPS is often used as the protocol. The server converts the voice to text and compares it with a database containing past fraud cases. This process utilizes speech recognition and natural language processing technologies.

[0632] The server incorporates a generative AI model to analyze voice information and detect fraudulent keywords and conversation patterns. If it determines that there is a high probability of fraud, the server immediately generates an alarm and notifies the device of this information. The notification is conveyed to the user through the device's push notification function or visual alert.

[0633] For example, when a user receives a suspicious phone solicitation, the system analyzes the conversation in real time and, if it determines the content to be fraudulent, immediately generates an alert and notifies family members and the police. This protects users from fraud in real time.

[0634] Furthermore, as an example of prompts to the generating AI model, questions such as "How can AI be used to issue real-time warnings if an elderly person is about to become a victim of bank deposit fraud?" can be asked. This system will protect users from the risk of fraud and enable them to live a safer life.

[0635] The flow of the specific processing in Example 1 will be explained using Figure 11.

[0636] Step 1:

[0637] The user activates a device equipped with an acoustic information acquisition mechanism and begins collecting audio data. The device uses a microphone to acquire ambient acoustic signals and encrypts this data within the device. The input is the ambient audio signals received by the microphone, and the output is the encrypted audio data. Specifically, when the user launches a dedicated application, the device automatically begins collecting audio.

[0638] Step 2:

[0639] The device transfers encrypted audio data to a server in the cloud using a secure protocol. The input is encrypted audio data stored on the device, and the output is encrypted data sent to the server. Specifically, data transmission uses the HTTPS protocol and is performed in the background without user intervention.

[0640] Step 3:

[0641] The server decrypts the received encrypted audio data and converts it to text using natural language processing techniques. The input is the encrypted audio data that arrives at the server, and the output is the audio information converted into text. Specifically, the server runs a speech recognition algorithm to accurately convert the audio into text.

[0642] Step 4:

[0643] The server analyzes transcribed audio data using machine learning analysis tools to assess the risk of fraud. The input is transcribed audio information, and the output is the fraud risk assessment result. Specifically, the server analyzes keywords and conversation patterns specific to fraud and scores the likelihood of fraud using a generative AI model.

[0644] Step 5:

[0645] If the server determines that there is a high probability of fraud, it immediately generates an alarm and notifies the terminal of this information. The input is the result of the fraud risk assessment, and the output is the generated alarm message. Specifically, the server creates the warning message in real time and sends it to the terminal via the notification system.

[0646] Step 6:

[0647] The device notifies the user of received alarms via push notifications and alert sounds. The input is the alarm message sent from the server, and the output is the warning notification to the user. Specifically, the device displays the warning message on the screen and simultaneously sounds an alert to draw the user's attention.

[0648] This allows the system to detect signs of fraud with high accuracy in real time, supporting a swift response.

[0649] (Application Example 1)

[0650] Next, we will explain Application Example 1. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0651] To address the vulnerability of the elderly to fraudulent verbal transactions such as bank deposit scams, technology is needed that analyzes voices in real time, quickly detects signs of fraud, and implements appropriate protective measures. Furthermore, there is a need for a system that is simple and automated for the elderly, providing a sense of security and safety.

[0652] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means.

[0653] In this invention, the server includes a voice acquisition mechanism for continuously acquiring user voice information, an information transmission mechanism for securely transferring voice information to a cloud infrastructure, and a machine learning analysis mechanism for evaluating the likelihood of fraud. This makes it possible to quickly detect signs of fraud, generate an alarm, and immediately notify the user and registered emergency contacts.

[0654] A "voice acquisition mechanism" is a device that plays the role of continuously collecting the user's voice information.

[0655] An "information transmission mechanism" is a device that transfers collected voice information to a cloud infrastructure via a secure communication protocol.

[0656] A "machine learning analysis mechanism" is a device that uses artificial intelligence technology to analyze collected audio information and evaluate the likelihood of fraud.

[0657] An "alarm generation mechanism" is a device that generates an alarm when the risk of fraud exceeds a certain threshold.

[0658] A "notification transmission mechanism" is a device that transmits generated alarms to pre-registered communication destinations.

[0659] An "instant notification mechanism" is a device that instantly notifies the user's mobile device or communication partner of an alarm.

[0660] An "information aggregation platform" is a database system that stores data on past fraud cases to help determine the likelihood of fraud.

[0661] The system realizing this invention begins with a terminal equipped with a voice acquisition mechanism that the user wears or carries. The terminal continuously collects ambient sound using a highly sensitive microphone and securely transmits this voice information to a cloud platform. In the cloud platform, the voice information is analyzed by a machine learning analysis mechanism to assess the likelihood of fraud. The analysis uses speech recognition technologies such as the Google Cloud Speech-to-Text API and a generative AI model that has learned the characteristics of fraud.

[0662] Based on the analysis results, the server generates an alert if the risk of fraud exceeds a certain threshold. When an alert is generated, it is sent to pre-registered contacts via a notification system. In addition, an instant notification system immediately notifies the user's mobile device, promptly alerting the user and their family.

[0663] For example, if a user receives a suspicious investment solicitation over the phone, the system collects the audio recording. If the recording is identified as containing dangerous keywords such as "Please transfer the money immediately," the system quickly generates an alert to inform the user of the situation. An alert is also immediately sent to pre-registered emergency contacts, such as family members' smartphones, allowing them to take immediate action to protect the user.

[0664] An example of a prompt for a generative AI model would be: "Analyze this voice conversation to see if it contains signs of fraud. The text is: 'Please provide your account number now...'"

[0665] This system makes it possible to prevent the risk of fraud targeting the elderly and provide a safe and secure living environment.

[0666] The flow of a specific process in Application Example 1 will be explained using Figure 12.

[0667] Step 1:

[0668] When the device is powered on, it uses its voice acquisition mechanism to collect ambient sounds. The input is ambient sound, and the output is audio data converted into a digital format. This data is temporarily stored within the device.

[0669] Step 2:

[0670] The device encrypts the collected audio data and transfers it to the cloud infrastructure using a secure protocol (e.g., HTTPS). The input is the encrypted audio data, and the output is the completion of the transmission to the cloud infrastructure.

[0671] Step 3:

[0672] The server receives audio data via a cloud infrastructure and performs analysis using a machine learning analysis mechanism. The input is the audio data received by the server, and the output is the transcribed conversation content and the results of the fraud probability assessment. For data processing, speech recognition technology (e.g., Google Cloud Speech-to-Text API) is used to convert speech to text, and then a generative AI model performs a fraud risk assessment.

[0673] Step 4:

[0674] Based on the analysis results, the server generates an alert if the fraud risk exceeds a certain threshold. The input is the assessment result of the likelihood of fraud, and the output is the trigger for the warning notification. Specifically, the alarm generation mechanism generates an alert when the risk determination threshold is exceeded.

[0675] Step 5:

[0676] The server utilizes a notification transmission mechanism to send generated alarms to pre-registered communication destinations and the user's mobile device. Inputs are the alarm notification and communication destination information, while output is the transmission of the alarm to recipients. Specifically, the immediate notification mechanism pushes the alarm to the user and simultaneously sends an alert to emergency contacts.

[0677] This series of processes makes it possible to detect fraud risks in real time and respond quickly.

[0678] Furthermore, an emotion engine that estimates the user's emotions may be incorporated. That is, the identification processing unit 290 may use the emotion identification model 59 to estimate the user's emotions and perform identification processing using the user's emotions.

[0679] This invention aims to improve the accuracy of fraud risk assessment by combining an emotion engine with a system that analyzes user voice data in real time and evaluates the likelihood of fraud. In addition to the conventional functions of collecting and analyzing user voice data, this system further improves fraud risk assessment and alert generation by recognizing the user's emotional state.

[0680] Users first use a smartphone with a dedicated app installed or a device equipped with a dedicated audio data collection system. The device continuously collects audio from the user's surroundings, encrypts it, and temporarily stores it. This audio data is then sent to a cloud server as needed.

[0681] Upon receiving audio data, the server first converts the audio into text using artificial intelligence analysis tools to detect keywords and patterns related to fraud. Next, it uses an emotion engine to extract the user's emotional state from the audio data. This emotional information is considered as one factor in assessing fraud risk.

[0682] For example, if the emotion engine determines that a user is in an excited state, it can prioritize the alert, considering the fraud risk to be higher than usual. In this way, emotional states are used as supplementary information to adjust the fraud risk assessment.

[0683] If the server determines that the situation is high-risk, it generates an alert using an alert generation mechanism and promptly notifies the user's device and pre-configured emergency contacts via a notification mechanism. By receiving the warning on their device, the user can take swift action to prevent fraud.

[0684] In this way, by incorporating an emotion engine, we can improve the accuracy of detecting fraudulent activity and provide a system that enhances user safety.

[0685] The following describes the processing flow.

[0686] Step 1:

[0687] The device constantly collects audio from the user's surroundings using its microphone. This audio data is encrypted for privacy protection and temporarily stored on the device.

[0688] Step 2:

[0689] The device encrypts the collected audio data and transfers it to the cloud server using a secure protocol. This ensures data confidentiality and prevents tampering.

[0690] Step 3:

[0691] Before analyzing the received audio data, the server converts it into text. This conversion process uses speech recognition technology.

[0692] Step 4:

[0693] The server processes the converted text using artificial intelligence analysis to detect keywords and patterns related to fraud. The detected information is used for an initial assessment of fraud risk.

[0694] Step 5:

[0695] The server uses an emotion engine to extract the user's emotional state from the audio data. Specifically, it analyzes the tone of voice and other factors to determine whether the user is excited, tense, or calm.

[0696] Step 6:

[0697] The server integrates the fraud assessment from step 4 and the emotional state from step 5 to reassess the fraud risk. If the emotional state is associated with a high risk, such as excitement, the risk assessment is adjusted.

[0698] Step 7:

[0699] If the server determines that there is a high risk of fraud, it will generate an alert using an alert generation mechanism. The alert will include the results of the fraud risk assessment and information on the emotional state.

[0700] Step 8:

[0701] The server quickly sends the generated alarm to the user's device and registered emergency contacts. Upon receiving the alarm, the device alerts the user using push notifications and an alert sound.

[0702] Step 9:

[0703] Users receive warnings from their devices and take appropriate action as needed. This allows them to take appropriate measures to prevent becoming a victim of fraud.

[0704] (Example 2)

[0705] Next, we will describe Example 2. In the following description, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0706] Fraudulent activities are becoming increasingly sophisticated, making detection extremely difficult. In particular, fraudsters employ psychological tricks to undermine victims' vigilance, limiting the effectiveness of traditional keyword-based detection methods. Furthermore, monitoring systems that fail to adequately consider users' emotional states can lead to inaccurate risk assessments. Therefore, there is a need for more accurate fraud risk assessment systems.

[0707] The identification process performed by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means.

[0708] In this invention, the server includes acoustic information collection means for continuously acquiring the user's voice information, information transmission means for securely transferring the voice information to an information processing device, and emotion analysis means for extracting emotional states from the voice information and including emotional information in the fraud risk assessment. This makes it possible to assess fraud risk with greater accuracy while taking into account the user's emotional state.

[0709] "User" refers to an individual or group that uses the system.

[0710] "Audio information" refers to the user's speech and surrounding sounds collected as acoustic data.

[0711] "Acoustic information acquisition means" refers to a device or method used to acquire the user's voice information.

[0712] An "information processing device" refers to a computer system used for data analysis, storage, and transfer.

[0713] "Information transmission means" refers to technical means for securely transferring collected audio information to a server or cloud.

[0714] "Machine learning analysis methods" refer to AI technologies and algorithms used to analyze voice information and assess the likelihood of fraud.

[0715] "Emotional analysis means" refers to a technology or algorithm for extracting the user's emotional state from voice information.

[0716] "Alert generation means" refers to a function that generates warnings or alerts when the risk of fraud exceeds a certain threshold.

[0717] "Notification means" refers to a communication means used to transmit generated alarms to users or pre-registered communication partners.

[0718] This invention combines a sentiment analysis function with a fraud risk assessment system that analyzes voice data in real time to enhance user safety. Specific embodiments are described below.

[0719] First, the user uses a mobile device or acoustic information collection device with a dedicated application installed that can collect voice information. The device uses a built-in microphone to continuously acquire the user's voice information and temporarily stores it as digital data. The data is protected from unauthorized access by others using encryption methods such as AES.

[0720] After the terminal collects voice information, it securely transmits the encrypted voice information to an information processing device in the cloud using an information transmission method. Communication is conducted via the HTTPS protocol, guaranteeing data integrity and confidentiality.

[0721] The server analyzes the received data. First, it uses machine learning analysis to convert the audio data into text data. For general text conversion, natural language processing models, known as speech recognition technology, are used. Furthermore, the server uses sentiment analysis to extract the user's emotional state from the audio data. Sentiment analysis employs algorithms that analyze the tone and rhythm of the voice.

[0722] If the server determines that there is a high risk of fraud, it will notify the user via an alarm generation system. This includes push notifications to the device and email notifications to registered communication partners. This allows users to take immediate action.

[0723] For example, when a user suddenly starts using a particular word frequently, emotional analysis can detect tension or excitement. As a result, the risk of fraud is assessed as higher than usual. An example of a prompt message that might be input to the generating AI model is, "Analyze the emotional signs related to fraud from this audio data."

[0724] By incorporating sentiment analysis in this way, we can provide a system that offers more accurate fraud risk assessments and supports users' safer lives.

[0725] The flow of the specific processing in Example 2 will be explained using Figure 13.

[0726] Step 1:

[0727] The terminal acquires audio information captured by the acoustic information collection device. The microphone detects all sounds the user makes and captures them as digital audio data. At this point, the input is an analog audio signal, and the output is digital audio data. The digitized audio information is temporarily stored on the terminal. During this time, the data is protected by AES encryption.

[0728] Step 2:

[0729] The terminal uses an information transmission method to send encrypted voice information to a server in the cloud. The input here is encrypted digital voice data, and the output is data securely transferred to the server. The HTTPS protocol ensures data integrity and confidentiality.

[0730] Step 3:

[0731] The server uses machine learning analysis to convert audio information into text. It receives digital audio data as input and outputs text data. By utilizing an acoustic model to convert audio into text, it creates a format suitable for analysis.

[0732] Step 4:

[0733] The server uses emotion analysis tools to analyze transcribed data and voice tone to extract the user's emotional state. The inputs are text data and digital voice data, while the output is data indicating the emotional state. The emotion analysis model detects excitement and tension based on the intonation and speed of the voice.

[0734] Step 5:

[0735] The server assesses the risk of fraud based on the analyzed data. The input consists of keyword detection results and sentiment information, while the output is a fraud risk assessment index. This assessment uses a generative AI model to determine the risk according to the prompt "Assess the fraud risk from this audio data."

[0736] Step 6:

[0737] The server issues an alert using an alarm generation mechanism if it determines that the fraud risk is high. The input is a fraud risk assessment index, and the output is the generated alarm. The alarm is pushed to the device and, in some cases, also notified to emergency contacts.

[0738] This series of processes allows users to respond quickly to fraud, improving security.

[0739] (Application Example 2)

[0740] Next, we will explain application example 2. In the following explanation, the data processing device 12 will be referred to as the "server" and the robot 414 as the "terminal".

[0741] In modern society, while online payments and voice-based transactions are increasing, fraudulent activities are also becoming more sophisticated. Therefore, there is a growing need for systems that can analyze voice data in real time and assess fraud risk with greater accuracy. In particular, there is a demand for systems that can interpret emotions from voice and assess fraud risk while taking the user's psychological state into account, but achieving this presents significant challenges.

[0742] The specific processing performed by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means.

[0743] In this invention, the server includes acoustic data collection means for continuously acquiring user voice information, data transmission means for encrypting the voice information and transferring it to a remote computer, artificial intelligence analysis means for converting the voice information into text and detecting words and patterns related to fraud, and an emotion analysis engine for recognizing emotional states and reflecting them in the fraud possibility assessment. This makes it possible to accurately assess the risk of fraud using emotion information extracted from voice data and appropriately notify the user of a warning.

[0744] "Acoustic data acquisition means" refers to devices or programs that continuously acquire ambient sound information, and have the function of collecting sound via smartphones or microphones.

[0745] "Data transmission means" refers to technologies and methods for encrypting collected audio information and securely transferring it to a remote computer.

[0746] "Artificial intelligence analysis means" refers to technologies and algorithms used to convert audio information into text and detect words and patterns related to fraud from that text.

[0747] A "sentiment analysis engine" refers to analytical technology that recognizes a user's emotional state based on voice information and reflects it in relation to the assessment of the risk of fraud.

[0748] An "alert generation method" refers to a device or method that generates an alert and notifies the user or registered contacts when the risk level of fraud exceeds a predetermined threshold.

[0749] "Notification method" refers to the technology or method used to send generated alarms to users or pre-registered contacts.

[0750] This invention is realized by equipping the user's terminal with acoustic data collection means. The terminal constantly collects audio from the user's surroundings, encrypts it, and temporarily stores it. This encrypted audio information is transferred to a cloud server in a secure manner using data transmission means as needed.

[0751] The server converts the received audio information into text using artificial intelligence analysis tools to detect words and patterns related to fraud. Software such as Google Cloud Speech-to-Text and AWS Transcribe can be used in this process. Furthermore, a sentiment analysis engine is utilized to identify the user's emotional state from the audio information. IBM Watson Tone Analyzer and Microsoft Text Analytics are suitable for this analysis.

[0752] The server combines data obtained from emotional states and voice information to assess the risk of fraud. If the risk of fraud exceeds a threshold, an alert generation system creates a warning and quickly sends the warning to the user's device or pre-registered contacts via a notification system.

[0753] For example, if an analysis of the audio generated when a user attempts to purchase an electronic device online detects an abnormal level of excitement, it is determined that the user is likely to be a fraud. In this situation, the user is warned to perform additional verification, such as by being prompted to enable two-factor authentication.

[0754] An example of a prompt is: "Detect whether the user is experiencing emotional distress during online payment and assess the likelihood of fraud."

[0755] The flow of a specific process in Application Example 2 will be explained using Figure 14.

[0756] Step 1:

[0757] The device continuously collects audio from the user's surroundings using acoustic data collection equipment. This audio data is acquired in real time as input and temporarily stored within the device. The data is encrypted and prepared for transmission to a cloud server in a secure state.

[0758] Step 2:

[0759] The terminal transfers encrypted voice information to a cloud server using a data transmission method. In this process, data is transferred over the network and prepared for reception by the server. The input is encrypted voice data, and the server receives the data as output.

[0760] Step 3:

[0761] The server uses artificial intelligence analysis to convert the received audio data into text. Here, Google Cloud Speech-to-Text is used as the data processing tool to convert audio data to text. The input to this process is audio data, and the output is text data in which fraud-related terms can be detected.

[0762] Step 4:

[0763] The server uses a sentiment analysis engine to determine the user's emotional state from text data. The input is the text data generated in step 3, and the output is information about the user's emotional state. IBM Watson Tone Analyzer is used for this sentiment assessment.

[0764] Step 5:

[0765] The server assesses the risk of fraud based on fraud-related terms and sentiment detected by artificial intelligence analysis. The input is text and sentiment data related to fraud risk, and the output is a risk assessment. If this risk level exceeds a threshold, an alert generation mechanism is triggered.

[0766] Step 6:

[0767] An alert is generated by the alert generation mechanism. This alert suggests a high probability of fraud and is ready to be sent as output to the user's device and registered contacts.

[0768] Step 7:

[0769] The server uses a notification system to send generated alarms to the user's device and designated contacts. The input is the generated alarm, and the output is a notification sent to the user and contacts. This allows the user to take immediate action.

[0770] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the controlled object 443 to output the result of the specific processing. The microphone 238 acquires audio indicating user input for the result of the specific processing. The control unit 46A transmits the audio data indicating user input acquired by the microphone 238 to the data processing unit 12. In the data processing unit 12, the specific processing unit 290 acquires the audio data.

[0771] Data generation model 58 is a type of so-called generative AI (Artificial Intelligence). One example of data generation model 58 is ChatGPT (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search) <url: https: gemini.google.com ?hl="ja">Examples of generative AI include the following. The data generation model 58 is obtained by performing deep learning on a neural network. The data generation model 58 is input with prompts containing instructions, and with inference data such as audio data representing speech, text data representing text, and image data representing images. The data generation model 58 infers from the input inference data according to the instructions indicated by the prompts, and outputs the inference results in data formats such as audio data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0772] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of this disclosure is not limited thereto, and the specific processing may also be performed by the robot 414.

[0773] Furthermore, the emotion identification model 59, acting as an emotion engine, may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to a specific mapping, which is an emotion map (see Figure 9). Similarly, the emotion identification model 59 may also determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[0774] Figure 9 shows an emotion map 400 in which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive the emotions are located. Further out of the concentric circles, emotions representing states and actions arising from mental states are located. Emotion is a concept that includes feelings and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions occurring in the brain are located. On the right side of the concentric circles, emotions that are generally induced by situational judgment are located. Above and below the concentric circles, emotions that are generally generated from reactions occurring in the brain and induced by situational judgment are located. In addition, the emotion of "pleasure" is located on the upper side of the concentric circles, and the emotion of "displeasure" is located on the lower side. Thus, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions arise, and emotions that are likely to occur simultaneously are mapped close together.

[0775] These emotions are distributed at the 3 o'clock position on the Emotion Map 400, and usually fluctuate between feelings of security and anxiety. In the right half of the Emotion Map 400, situational awareness takes precedence over internal feelings, resulting in a calm impression.

[0776] The inside of the Emotion Map 400 represents inner thoughts, while the outside represents actions. Therefore, the further you go from the outside of the Emotion Map 400, the more visible (expressed in actions) your emotions become.

[0777] Here, human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. Similarly, in robots, cars, motorcycles, etc., emotions can be created based on various balances, such as posture and battery level. When these balances deviate from the ideal, it results in discomfort, and when they approach the ideal, it results in pleasure. The emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on a system for analyzing brain physiological signals of speech emotion recognition and emotion, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map contains emotions belonging to a region called "response," where sensation is dominant. The right half of the emotion map contains emotions belonging to a region called "situation," where situational awareness is dominant.

[0778] The emotion map defines two emotions that promote learning. One is the emotion around the middle of the negative "repentance" and "reflection" on the situation side. In other words, it is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the emotion around the positive "desire" on the reaction side. In other words, it is when the robot has positive feelings such as "I want more" or "I want to know more."

[0779] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​representing each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple training data sets, which are combinations of user input and emotion values ​​representing each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions located close together have similar values, as shown in the emotion map 900 in Figure 10. Figure 10 shows an example where multiple emotions such as "reassured," "calm," and "confident" have similar emotion values.

[0780] The above description primarily focuses on the functions of the data processing device 12 in relation to this disclosure. However, the system related to this disclosure is not necessarily implemented on a server. The system related to this disclosure may be implemented as a general information processing system. This disclosure may be implemented, for example, as a software program that runs on a personal computer or as an application that runs on a smartphone. The method related to this disclosure may be provided to users in SaaS (Software as a Service) format.

[0781] In the above embodiment, an example was given in which a specific process is performed by a single computer 22. However, the technology of this disclosure is not limited thereto, and a distributed processing of the specific process may be performed by multiple computers, including computer 22. For example, a data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to the input data.

[0782] In the above embodiment, an example was given in which the specific processing program 56 is stored in the storage 32, but the technology of this disclosure is not limited thereto. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-temporary storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-temporary storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes specific processing according to the specific processing program 56.

[0783] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[0784] Furthermore, it is not necessary to store the entirety of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store the entirety of the specific processing program 56 in the storage 32; it is acceptable to store only a portion of the specific processing program 56.

[0785] The following types of processors can be used as hardware resources to perform specific processing. Examples of processors include a CPU, a general-purpose processor that functions as a hardware resource to perform specific processing by executing software, i.e., a program. Other examples of processors include dedicated electrical circuits, such as FPGAs (Field-Programmable Gate Arrays), PLDs (Programmable Logic Devices), or ASICs (Application Specific Integrated Circuits), which have circuit configurations specifically designed to perform specific processing. All of these processors have built-in or connected memory, and all of them perform specific processing by using memory.

[0786] The hardware resource that performs a specific process may consist of one of these various processors, or it may consist of a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Alternatively, the hardware resource that performs a specific process may consist of a single processor.

[0787] Examples of configurations using a single processor include, firstly, a configuration in which one or more CPUs and software are combined to form a single processor, and this processor functions as a hardware resource that performs a specific process. Secondly, there is a configuration using a processor that realizes the functions of the entire system, including multiple hardware resources that perform a specific process, on a single IC chip, as exemplified by SoCs (System-on-a-chip). In this way, a specific process is realized using one or more of the above types of processors as hardware resources.

[0788] Furthermore, the hardware structure of these various processors can more specifically utilize electrical circuits that combine circuit elements such as semiconductor devices. Also, the specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps can be deleted, new steps added, or the processing order rearranged, as long as it does not deviate from the main purpose.

[0789] The descriptions and illustrations presented above are detailed explanations of the technical aspects of this disclosure and are merely examples of the technical aspects. For example, the above descriptions of the structure, function, operation, and effect are examples of the structure, function, operation, and effect of the technical aspects of this disclosure. Therefore, it goes without saying that you may delete unnecessary parts, add new elements, or replace elements in the descriptions and illustrations presented above, as long as you do not deviate from the essence of the technical aspects of this disclosure. Furthermore, in order to avoid confusion and facilitate understanding of the technical aspects of this disclosure, explanations of common technical knowledge and the like that do not require special explanation to enable the implementation of the technical aspects of this disclosure have been omitted from the descriptions and illustrations presented above.

[0790] All documents, patent applications, and technical standards described herein are incorporated by reference to the same extent as if each individual document, patent application, and technical standard were specifically and individually noted to be incorporated by reference.

[0791] The following is further disclosed regarding the embodiments described above.

[0792] (Claim 1)

[0793] A means for collecting acoustic data to continuously acquire user voice data,

[0794] A data transmission means for securely transferring the aforementioned audio data to the cloud,

[0795] An artificial intelligence analysis means for analyzing the aforementioned audio data and evaluating the possibility of fraud,

[0796] An alert generation method for generating a warning when the risk of fraud exceeds a certain threshold,

[0797] A notification means for sending the aforementioned alarm to a pre-registered communication destination,

[0798] A system that includes this.

[0799] (Claim 2)

[0800] The system according to claim 1, which has a function that can stop collecting audio data based on user actions.

[0801] (Claim 3)

[0802] The system according to claim 1, which determines the likelihood of fraud by linking with a database based on past fraud cases.

[0803] "Example 1"

[0804] (Claim 1)

[0805] A means for acquiring acoustic information to continuously collect audio information,

[0806] Information transmission means for securely transferring the aforementioned voice information to a communication network,

[0807] A machine learning analysis method for analyzing the aforementioned audio information and evaluating the possibility of fraud with high accuracy,

[0808] A warning generation method for generating an alert when the risk of fraud exceeds a certain threshold,

[0809] A notification means for sending the aforementioned alarm to a pre-registered communication destination,

[0810] A means for converting the aforementioned audio information into text using natural language processing technology,

[0811] A means of displaying information to warn the user,

[0812] A system that includes this.

[0813] (Claim 2)

[0814] The system according to claim 1, further comprising a function that can stop the collection of voice information based on the user's actions.

[0815] (Claim 3)

[0816] The system according to claim 1, which determines the likelihood of fraud in conjunction with information sources based on past fraud cases.

[0817] "Application Example 1"

[0818] (Claim 1)

[0819] A voice acquisition mechanism for continuously acquiring user voice information,

[0820] An information transmission mechanism for securely transferring the aforementioned audio information to a cloud infrastructure,

[0821] A machine learning analysis mechanism for analyzing the aforementioned audio information and evaluating the possibility of fraud,

[0822] An alarm generation mechanism for generating an alert when the risk of fraud exceeds a certain threshold,

[0823] A notification transmission mechanism for sending the aforementioned alarm to a pre-registered communication destination,

[0824] An immediate notification mechanism for immediately notifying the user's mobile device or communication partner of the aforementioned alarm,

[0825] A system that includes this.

[0826] (Claim 2)

[0827] The system according to claim 1, further comprising a function that can stop the collection of voice information based on the user's actions.

[0828] (Claim 3)

[0829] The system according to claim 1, which determines the likelihood of fraud in conjunction with an information aggregation platform based on past fraud cases.

[0830] "Example 2 of combining an emotion engine"

[0831] (Claim 1)

[0832] A means for collecting acoustic information to continuously acquire the user's voice information,

[0833] Information transmission means for securely transferring the aforementioned audio information to an information processing device,

[0834] A machine learning analysis means for analyzing the aforementioned audio information and evaluating the possibility of fraud,

[0835] A means for extracting emotional states from the aforementioned audio information and for including emotional information in the assessment of fraud risk,

[0836] An alarm generation means for generating an alarm when the risk of fraud exceeds a certain standard,

[0837] A notification means for transmitting the aforementioned alarm to a pre-registered communication partner,

[0838] A system that includes this.

[0839] (Claim 2)

[0840] The system according to claim 1, further comprising a function that can stop the collection of voice information based on the user's actions.

[0841] (Claim 3)

[0842] The system according to claim 1, which determines the likelihood of fraud by linking it with information accumulated from past fraud cases.

[0843] "Application example 2 when combining with an emotional engine"

[0844] (Claim 1)

[0845] A means for collecting acoustic data to continuously acquire user voice information,

[0846] A data transmission means for encrypting the aforementioned audio information and transferring it to a remote computer,

[0847] An artificial intelligence analysis means that converts the aforementioned audio information into text and detects words and patterns related to fraud,

[0848] A sentiment analysis engine to recognize emotional states and reflect them in the assessment of potential fraud,

[0849] An alert generation method for issuing a warning when the risk level of fraud exceeds a certain threshold,

[0850] A notification means for sending the aforementioned alarm to a pre-registered contact,

[0851] A system that includes this.

[0852] (Claim 2)

[0853] The system according to claim 1, which has a function that can stop collecting voice information depending on the user's activity.

[0854] (Claim 3)

[0855] The system according to claim 1, which determines the likelihood of fraud in cooperation with an information aggregation device related to past fraud cases. [Explanation of Symbols]

[0856] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Devices 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robots< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for collecting acoustic data to continuously acquire user voice data, A data transmission means for securely transferring the aforementioned audio data to the cloud, An artificial intelligence analysis means for analyzing the aforementioned audio data and evaluating the possibility of fraud, An alert generation method for generating a warning when the risk of fraud exceeds a certain threshold, A notification means for sending the aforementioned alarm to a pre-registered communication destination, A system that includes this.

2. The system according to claim 1, which has a function that can stop collecting audio data based on user actions.

3. The system according to claim 1, which determines the likelihood of fraud by linking with a database based on past fraud cases.

Images