System

Abstract

To provide a system capable of preventing a fraudulent behavior.SOLUTION: This system comprises: means for collecting voices; conversion means for converting the collected voices into text using a natural language processing feature; detection means for detecting whether the collected voices contain a conversation of possible fraud by inputting the converted text and a prompt sentence instructing determination as to whether the text contains a conversion of possible fraud to a generative AI model; and issuance means for automatically issuing an alert to a registered contact address when it is detected that the conversation of possible fraud is contained.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Currently, frauds such as "it's me" frauds are on the rise, and many of the victims are elderly people. There are many different types of frauds, and it is difficult for victims to recognize them. Therefore, there is a need for methods to prevent frauds before they occur. [Means for solving the problem]

[0005] This invention collects audio and analyzes it to detect conversations that may be fraudulent. If a conversation that may be fraudulent is detected, an alert is issued. Audio is collected using a dedicated device, and conversations that may be fraudulent are detected using chat generation AI. This makes it possible to prevent fraudulent acts before they occur. [Brief explanation of the drawings]

[0006] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 2 is a sequence diagram showing a flow of processing in the data processing system according to the first embodiment of the first form example. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Embodiment 1. [Figure 13] FIG. 10 is a sequence diagram showing a processing flow of a data processing system in a second embodiment of the second form example. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Embodiment Example 2. [Figure 15] FIG. 10 is a sequence diagram showing the flow of processing in a data processing system according to a third embodiment of the third embodiment. [Figure 16]FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Embodiment 3. [Figure 17] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the first embodiment of the first form example when an emotion engine is combined. [Figure 18] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1 of Form Example 1 when an emotion engine is combined. [Figure 19] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the second embodiment of the second form example when an emotion engine is combined. [Figure 20] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 of Form Example 2 when an emotion engine is combined. [Figure 21] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in the third embodiment of the third form example when an emotion engine is combined. [Figure 22] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 3 of Form Example 3 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0007] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0008] First, the terms used in the following description will be explained.

[0009] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, the processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), or a TPU (TENSOR PROCESSING UNIT (registered trademark)).

[0010] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0011] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0012] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0013] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0014] [First embodiment]

[0015] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0016] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0017] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0018] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0019] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0020] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0021] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0022] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0023] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0024] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0025] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0026] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0027] "Example 1"

[0028] In one embodiment of the present invention, voice is collected using a dedicated terminal. This terminal is installed in a place frequently used by the elderly, such as a living room or bedroom. The terminal constantly monitors the elderly's conversations and collects their voice.

[0029] "Example 2"

[0030] The collected audio is analyzed using chat generation AI. The chat generation AI uses natural language processing technology to convert the audio into text and determine whether the text contains conversations that could be fraudulent. For example, if the text contains a phrase such as "I want you to send me money," it is determined to be a potential scam.

[0031] "Example 3"

[0032] If the device detects a potentially fraudulent conversation, it will issue an alert. The alert can be issued in various forms, such as a voice message from the device or an automated call to registered contacts (e.g., family members or the police). This makes it possible to prevent fraud before it occurs.

[0033] The processing flow of each embodiment will be described below.

[0034] "Example 1"

[0035] Step 1: Install a dedicated device in a location frequently used by the elderly, such as the living room or bedroom.

[0036] Step 2: The device constantly monitors the elderly person's conversations and collects their audio.

[0037] "Example 2"

[0038] Step 1: The collected audio is analyzed using chat generation AI.

[0039] Step 2: The chat generation AI uses natural language processing technology to convert speech into text.

[0040] Step 3: Determine if the text contains potentially fraudulent conversations. For example, if it contains a phrase like "please send me money," it is deemed potentially fraudulent.

[0041] "Example 3"

[0042] Step 1: If the device detects a potentially fraudulent conversation, it will issue an alert.

[0043] Step 2: The alert can take various forms, such as a voice message from the device or an automated call to registered contacts (e.g., family members or the police).

[0044] Example 1

[0045] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0046] There is a need to reduce the risk of elderly people becoming victims of fraud and improve their safety in their daily lives. However, conventional systems do not adequately collect and analyze voice data, making it difficult to detect potentially fraudulent conversations in real time. Furthermore, there is a lack of a way for users to easily check the analysis results. This has prevented the realization of effective measures to ensure the safety of elderly people.

[0047] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0048] In this invention, the server includes means for collecting voice, means for analyzing the collected voice and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, means for transmitting the collected voice data to the server, means for the server to analyze the voice data, means for saving the analysis results, and means for the user to check the analysis results. This makes it possible to monitor elderly people's conversations in real time, quickly detect conversations that may be fraudulent, and allow the user to easily check the analysis results.

[0049] The "means for collecting voice" is a dedicated terminal equipped with a highly sensitive microphone, which is a device that constantly monitors the elderly person's conversations and collects voice data.

[0050] The "means of analyzing collected audio to detect potentially fraudulent conversations" refers to a system that uses a generative AI model to convert audio data into text, analyzes the content of that text, and detects potentially fraudulent conversations.

[0051] The "means for issuing an alert when a potentially fraudulent conversation is detected" is a notification system for issuing a warning to the user when a potentially fraudulent conversation is detected.

[0052] The "means for transmitting collected voice data to a server" refers to a function in which the terminal connects to a server via the Internet and encrypts and transmits the collected voice data.

[0053] "Means by which the server analyzes voice data" refers to the process by which the server converts the voice data it receives into text using a generative AI model and analyzes the content.

[0054] The "means for saving the analysis results" is a function that allows the server to save the analysis results in a database, allowing users to easily search for the information they need later.

[0055] The "means for users to check the analysis results" refers to a system that allows users to access the server through a dedicated web interface or mobile app and view the analysis results.

[0056] MODE FOR CARRYING OUT THE INVENTION

[0057] This invention is a system for monitoring elderly people's conversations and detecting potentially fraudulent conversations. The system includes a dedicated terminal, a server, a generative AI model, and a user interface.

[0058] 1. Hardware and Software Configuration

[0059] Terminal

[0060] The device is equipped with a highly sensitive microphone and is installed in places frequently used by elderly people, such as living rooms and bedrooms, and has the function of temporarily storing collected voice data in local storage.

[0061] server

[0062] The server receives voice data from the device via the internet, has a generative AI model (e.g., OpenAI's GPT-4®) installed on it, converts the voice data into text, and analyzes its content.

[0063] Generative AI Models

[0064] The generative AI model includes speech recognition technology to convert voice data into text and natural language processing technology to analyze the content of the text, thereby detecting potentially fraudulent conversations.

[0065] User Interface

[0066] Users can access the server and check the analysis results through a dedicated web interface or mobile app. The interface is designed to be easy for users to use.

[0067] 2. Data processing and calculation

[0068] Audio data collection

[0069] The device constantly monitors the elderly person's conversations and collects voice data. For example, if an elderly person is talking to a friend, such as "The weather is nice today," the voice is collected in real time.

[0070] Sending audio data

[0071] The collected voice data is sent from the device to the server at regular intervals (for example, every hour). The data is encrypted before being sent.

[0072] Analysis of audio data

[0073] The server converts the received voice data into text using a generative AI model. This conversion is performed using speech recognition technology. The generative AI model then analyzes the content of the text and extracts important information and unusual patterns. For example, the text "The weather is nice today" is generated and its content is analyzed.

[0074] Saving analysis results

[0075] The server stores the analysis results in a database, including the original audio data, the converted text, and the analysis results, allowing users to easily search for the information they need later.

[0076] Checking the analysis results

[0077] Users can access the server through a dedicated web interface or mobile app and check the analysis results. For example, if a family member wants to check the content of an elderly person's recent conversations, they can log in to the web interface and view the analysis result, such as "The weather is nice today."

[0078] 3. Examples and prompts

[0079] Specific examples

[0080] An elderly person is having a conversation with a friend in the living room, and the content of this conversation is collected by a device and analyzed using a generative AI model.

[0081] Prompt Sentence Examples

[0082] "Collect audio data of an elderly person having a conversation with a friend in their living room, and use a generative AI model to convert the conversation into text."

[0083] The above is an embodiment of the present invention.

[0084] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0085] Step 1:

[0086] The device collects the audio.

[0087] Input: Elderly people's conversation voice

[0088] How it works: A highly sensitive microphone installed in the device collects the elderly person's conversation in real time. For example, if an elderly person says to a friend, "The weather is nice today," the voice is collected.

[0089] Output: Collected audio data

[0090] Step 2:

[0091] The device stores the audio data in local storage.

[0092] Input: Collected audio data

[0093] Specific operation: The collected voice data is temporarily stored in the local storage of the device.

[0094] Output: Audio data saved in local storage

[0095] Step 3:

[0096] The terminal transmits the voice data to the server.

[0097] Input: Audio data stored in local storage

[0098] Specific operation: At regular intervals (for example, every hour), the device connects to a server via the Internet and encrypts and transmits the collected voice data.

[0099] Output: Audio data sent to the server

[0100] Step 4:

[0101] The server analyzes the audio data.

[0102] Input: Audio data sent to the server

[0103] How it works: The server converts the received voice data into text using a generative AI model (e.g., OpenAI's GPT-4). This conversion uses speech recognition technology. The generative AI model then analyzes the content of the text to extract important information and unusual patterns. For example, the text "The weather is nice today" is generated, and its content is analyzed.

[0104] Output: Parsed text data and analysis results

[0105] Step 5:

[0106] The server stores the analysis results.

[0107] Input: Parsed text data and analysis results

[0108] Specific operation: The server saves the analysis results in a database. The saved data includes the original voice data, the converted text, and the analysis results, allowing users to easily search for the information they need later.

[0109] Output: Analysis results stored in a database

[0110] Step 6:

[0111] The user checks the analysis results.

[0112] Input: Analysis results stored in the database

[0113] Specific operation: Users access the server through a dedicated web interface or mobile app to check the analysis results. For example, if a family member wants to check the content of an elderly person's recent conversations, they can log in to the web interface and view the analysis result, such as "The weather is nice today."

[0114] Output: Analysis results confirmed by the user

[0115] (Application example 1)

[0116] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0117] In the living spaces of elderly people, there is a need for a system that can quickly detect abnormal voice patterns (such as cries for help or the sound of someone falling) and take appropriate action. However, conventional systems have low accuracy in analyzing voice data, and there is a risk that abnormalities will be overlooked. In addition, there is a lack of means to quickly issue an alert when an abnormality is detected, which can delay emergency response. To solve these issues, a system that can perform highly accurate voice analysis and issue alerts quickly is needed.

[0118] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0119] In this invention, the server includes means for collecting voice, means for analyzing the collected voice and detecting abnormal voice patterns, and means for issuing an alert when an abnormal voice pattern is detected. This makes it possible to quickly detect abnormal voice patterns in the elderly person's living space and take appropriate measures.

[0120] The "means for collecting audio" is a dedicated terminal that is installed in the elderly person's living space and is used to constantly collect surrounding audio.

[0121] "Means for analyzing collected voice data and detecting abnormal voice patterns" refers to a function that analyzes collected voice data using a generative AI model to detect abnormal voice patterns such as calls for help or the sound of a fall.

[0122] "Means for issuing an alert when an abnormal voice pattern is detected" is a function that sends an alert to a device such as a smartphone when an abnormal voice pattern is detected, prompting a prompt response.

[0123] A "generative AI model" is an artificial intelligence model used to analyze voice data and detect anomalous voice patterns with high accuracy.

[0124] A "dedicated terminal" is a specialized device that is installed in the elderly person's living space and is used to collect voice.

[0125] The system for implementing this invention consists of a dedicated terminal installed in the elderly person's living space, a voice analysis system using a generative AI model, and a smartphone application that issues an alert when an abnormality is detected.

[0126] Hardware and Software Configuration

[0127] 1. Dedicated terminal: This device is installed in the elderly person's living space (living room, bedroom, etc.) and constantly collects surrounding sounds. This terminal is equipped with a microphone and a communication module for transmitting voice data to a server.

[0128] 2. Generative AI model: A voice analysis system that runs on a server. It is built using machine learning libraries such as TENSORFLOW (registered trademark) and Keras, and analyzes collected voice data to detect abnormal voice patterns (such as calls for help or the sound of someone falling).

[0129] 3. Smartphone application: This application sends an alert to the user when an abnormal voice pattern is detected. When the application receives an alert, it notifies the user and prompts them to take action.

[0130] Processing flow

[0131] 1. Collection of voice data: A dedicated device collects voice data in the elderly person's living space and transmits it to a server in real time.

[0132] 2. Voice data analysis: A generative AI model on the server analyzes the collected voice data and detects anomalous voice patterns.

[0133] 3. Alerting: If an abnormal voice pattern is detected, the server sends an alert to the smartphone application, which displays a notification to the user, prompting them to take immediate action.

[0134] Specific examples

[0135] For example, if an elderly person falls in the middle of the night, a dedicated device will collect the sound and the generative AI model will recognize it as the "sound of a fall." If an abnormality is detected, the server will send an alert to the user's smartphone saying, "An elderly person has fallen."

[0136] Prompt Sentence Examples

[0137] "Voice data is collected from a dedicated device installed in the elderly's living space and analyzed using a generative AI model. If an abnormal voice pattern (such as a cry for help or the sound of a fall) is detected, an alert is sent to the smartphone."

[0138] In this way, it becomes possible to monitor the safety of elderly people in real time and respond quickly if an abnormality occurs.

[0139] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0140] Step 1:

[0141] The device collects audio in the elderly person's living space. The device uses a microphone to capture ambient audio in real time and converts the data into a digital format. The input is ambient audio, and the output is digital audio data.

[0142] Step 2:

[0143] The terminal transmits the collected digital voice data to the server. The terminal uses a communication module to transmit the collected voice data to the server via the Internet. The input is the digital voice data, and the output is the voice data transmitted to the server.

[0144] Step 3:

[0145] The server inputs the received voice data into a generative AI model. The server uses a generative AI model built using TensorFlow and Keras to analyze the voice data. The input is the voice data sent to the server, and the output is the analysis result.

[0146] Step 4:

[0147] The server detects abnormal voice patterns based on the analysis results of the generative AI model. The server evaluates the analysis results and determines whether they contain abnormal voice patterns such as calls for help or the sound of a fall. The input is the analysis results of the generative AI model, and the output is the anomaly detection results.

[0148] Step 5:

[0149] If the server detects an abnormal voice pattern, it sends an alert to the smartphone application. The server generates an alert message based on the anomaly detection results and sends it to the smartphone. The input is the anomaly detection result, and the output is the alert message.

[0150] Step 6:

[0151] A user receives an alert on a smartphone application. The application notifies the user of the received alert message and prompts them to take action. The input is the alert message, and the output is a notification to the user.

[0152] Example 2

[0153] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0154] Conventional voice analysis systems have difficulty detecting potentially fraudulent conversations in real time, putting users at high risk of becoming victims of fraud. Furthermore, the accuracy of voice data analysis is low, resulting in frequent false positives and false negatives. This creates a need for a highly reliable system to ensure user safety.

[0155] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0156] In this invention, the server includes means for receiving and storing voice data, means for converting the voice data into text, and means for analyzing the text to detect conversations that may be fraudulent, thereby enabling highly accurate analysis of voice data and real-time detection of conversations that may be fraudulent.

[0157] "Audio collection means" means a device or function for recording a user's speech and converting it into digital form.

[0158] The "means for transmitting collected voice to a server" is a device or function for transmitting voice data recorded by the terminal to a server via the Internet.

[0159] The "means for the server to receive and store voice data" refers to a device or function that allows the server to receive voice data sent from the terminal and store it in a specific folder or database.

[0160] The "means for the server to convert voice data into text" refers to a device or function for converting voice data received by the server into text format using natural language processing technology.

[0161] "Means for the server to analyze text and detect potentially fraudulent conversations" refers to a device or function that allows the server to analyze text data and detect specific phrases or patterns to determine whether there is a possibility of fraud.

[0162] "Means for issuing an alert when a potentially fraudulent conversation is detected" refers to a device or function for sending a warning message or notification to a user when the server detects a potentially fraudulent conversation.

[0163] A "dedicated terminal" is an electronic device specialized for collecting and transmitting audio to a server.

[0164] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to convert voice data into text and analyze the text data.

[0165] MODE FOR CARRYING OUT THE INVENTION

[0166] The present invention relates to a system for collecting voice data and detecting potentially fraudulent conversations. A specific embodiment of this system will be described below.

[0167] Audio data collection

[0168] The user uses the terminal to collect voice data. The terminal is equipped with a microphone that records the user's voice. For example, when the user says, "I want to send money," the voice is recorded on the terminal.

[0169] Sending audio data

[0170] The device transmits the collected voice data to a server. This requires an internet connection. Specifically, the device converts the voice data into a digital format and sends it to the server using an HTTP request.

[0171] Receiving and storing audio data

[0172] The server receives the voice data sent from the device. The server temporarily stores the received voice data and prepares it for analysis. For example, the server stores the voice data in a specific folder.

[0173] Converting audio data to text

[0174] The server passes the saved voice data to the chat generation AI, which uses natural language processing technology such as OpenAI's GPT-4. The server inputs the voice data into the AI, which then converts the voice into text.

[0175] Text Analysis

[0176] The server analyzes the converted text to determine whether it is potentially fraudulent. Specifically, the server searches for specific phrases in the text (e.g., "Please send me money") and evaluates whether it is potentially fraudulent.

[0177] Notification of results

[0178] The server sends the analysis results to the device, where the user can check the results. For example, the server sends a message to the device saying "Possible fraud," and the device displays the message to the user.

[0179] Specific examples

[0180] Example 1: Possible fraud

[0181] The user records a voice message saying "I want to send money," and the device sends the voice message to the server. The server converts the voice message into text and determines that it may be fraudulent. As a result, the device displays a message saying, "This may be fraudulent."

[0182] Example 2: No possibility of fraud

[0183] The user records "Hello, how are you?" and the device sends the audio to the server. The server converts the audio into text and determines that there is no possibility of fraud. As a result, the device displays a message saying "No possibility of fraud."

[0184] Prompt Sentence Examples

[0185] Here are some example prompts to input to a generative AI model:

[0186] Convert the audio data into text and determine whether it is likely to be fraudulent. For example, if it contains a phrase like "I want you to send me money," determine that it is likely to be fraudulent.

[0187] Using this prompt, the generative AI model can properly analyze the voice data and determine the likelihood of fraud.

[0188] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0189] Step 1:

[0190] A user uses a terminal to collect voice data. The user presses the record button on the terminal and says, "I want to send money." The terminal records the voice and converts it into a digital format. The input is the user's voice, and the output is digital voice data.

[0191] Step 2:

[0192] The device sends the collected voice data to the server. The device encodes the voice data in a digital format and sends it to the server using an HTTP request. The input is the digital voice data, and the output is the voice data sent to the server.

[0193] Step 3:

[0194] The server receives and saves the audio data sent from the device. The server receives an HTTP request and saves the audio data in a specific folder. The input is the audio data sent from the device, and the output is the audio data saved on the server.

[0195] Step 4:

[0196] The server passes the voice data it has saved to the chat generation AI. The server inputs the voice data into the AI, which then converts the voice into text. The input is the voice data saved on the server, and the output is text data.

[0197] Step 5:

[0198] The server analyzes the converted text and determines whether it is likely to be fraudulent. The server searches for specific phrases in the text (e.g., "Please send me money") and evaluates whether it is likely to be fraudulent. The input is text data, and the output is the evaluation result regarding the likelihood of fraud.

[0199] Step 6:

[0200] The server sends the analysis results to the terminal. The server sends a message saying "Possible fraud" to the terminal, and the terminal displays the message to the user. The input is the evaluation result regarding the possibility of fraud, and the output is a warning message displayed to the user.

[0201] (Application example 2)

[0202] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0203] In recent years, fraud has become more sophisticated, with voice fraud in particular on the rise. This has increased the risk of many people becoming victims of fraud. Conventional fraud detection systems have the challenge of analyzing voice in real time and quickly determining whether a fraud is likely. Furthermore, there is a need for a system that can monitor the voice calls received by users and surrounding conversations in real time, and immediately issue a warning if there is a possibility of fraud.

[0204] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice, means for analyzing the collected voice and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, means for converting the collected voice into text, means for analyzing the converted text using a generative AI model, and means for inputting a prompt sentence into the generative AI model to determine the possibility of fraud. This makes it possible to monitor voice calls received by the user and surrounding conversations in real time and to immediately issue an alert if there is a possibility of fraud.

[0205] "Means for collecting audio" refers to devices or software for acquiring audio as digital data.

[0206] "Means for analyzing collected audio and detecting potentially fraudulent conversations" refers to devices or software that analyze acquired audio data and determine whether its content constitutes fraudulent activity.

[0207] "Means for issuing an alert when a potentially fraudulent conversation is detected" refers to a device or software that issues a warning to the user when a potentially fraudulent conversation is detected.

[0208] "Means for converting collected voice into text" refers to devices or software for converting voice data into text data.

[0209] "Means for analyzing converted text using a generative AI model" refers to devices or software that analyze text data using a generative AI model and understand its content.

[0210] "Means for inputting prompt sentences into a generative AI model to determine the possibility of fraud" refers to devices or software that input specific prompt sentences into a generative AI model and determine the possibility of fraud based on the response.

[0211] A system for implementing this invention includes means for collecting audio, means for analyzing the collected audio to detect potentially fraudulent conversations, means for issuing an alert when potentially fraudulent conversations are detected, means for converting the collected audio into text, means for analyzing the converted text using a generative AI model, and means for inputting a prompt sentence into the generative AI model to determine the likelihood of fraud.

[0212] Hardware and Software Configuration

[0213] 1. Audio collection method:

[0214] Hardware: Using the microphone in your smartphone or smart glasses.

[0215] Software: For audio collection, we use the speech_recognition library.

[0216] 2. Speech-to-text methods:

[0217] Hardware: Server or cloud service.

[0218] Software: Uses the Google® Cloud Speech-to-Text API to convert collected audio into text in real time.

[0219] 3. Analysis methods using generative AI models:

[0220] Hardware: Server or cloud service.

[0221] Software: OpenAI's GPT-4 is used to analyze the converted text.

[0222] 4. Enter prompt text to determine potential fraud:

[0223] Hardware: Server or cloud service.

[0224] Software: Using OpenAI's API, specific prompts are fed into a generative AI model, and the response is used to determine the likelihood of fraud.

[0225] 5. Alert generation method:

[0226] Hardware: Smartphones and smart glasses.

[0227] Software: Applications that notify users of potential fraud.

[0228] Specific examples

[0229] For example, if a user receives a call containing the phrase "send me money," the system works as follows:

[0230] 1. Audio collection: The smartphone microphone collects audio.

[0231] 2. Speech-to-text conversion: The collected audio is converted to text using the Google Cloud Speech-to-Text API.

[0232] 3. Analysis using a generative AI model: The converted text is analyzed using OpenAI's GPT-4.

[0233] 4. Enter the prompt: Enter the following prompt into the generative AI model:

[0234] "Please determine if the following text is a potential scam: Send me money."

[0235] 5. Determining the likelihood of fraud: Determine the likelihood of fraud based on the response of the generative AI model.

[0236] 6. Alert: If a potential fraud attempt is detected, a warning notification will be sent to your smartphone.

[0237] The system allows users to detect potentially fraudulent conversations in real time and respond quickly.

[0238] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0239] Step 1:

[0240] A user collects audio using a smartphone or smart glasses. The input is ambient sound or phone conversation audio, and the output is digital audio data. Specifically, the device's microphone captures the audio and stores it as audio data.

[0241] Step 2:

[0242] The terminal sends the collected voice data to the server. The input is digital voice data, and the output is the voice data sent to the server. Specifically, the terminal's communication module uploads the voice data to the server via the Internet.

[0243] Step 3:

[0244] The server converts the voice data into text using the Google Cloud Speech-to-Text API. The input is digital voice data, and the output is text data. Specifically, the server sends the voice data to the API and receives the returned text data.

[0245] Step 4:

[0246] The server analyzes the converted text data using a generative AI model (OpenAI's GPT-4). The input is text data, and the output is the analysis result. Specifically, the server inputs the text data into the generative AI model and obtains the analysis result.

[0247] Step 5:

[0248] The server inputs a prompt sentence into the generative AI model to determine the likelihood of fraud. The input is the prompt sentence and text data, and the output is the judgment result regarding the likelihood of fraud. Specifically, the server inputs the following prompt sentence into the generative AI model:

[0249] "Please determine if the following text is a potential scam: Send me money."

[0250] Step 6:

[0251] If the server determines that there is a possibility of fraud, it issues an alert to the terminal. The input is the result of the judgment regarding the possibility of fraud, and the output is a warning notification to the user. Specifically, the server sends a notification to the terminal based on the judgment result, and the terminal displays a warning to the user.

[0252] Step 7:

[0253] The user receives the warning and takes the necessary action. The input is the warning notification, and the output is the user's response action. Specifically, the user checks the warning and takes appropriate action against the potentially fraudulent conversation.

[0254] Example 3

[0255] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0256] In recent years, fraud has become more sophisticated, with an increase in fraudulent activities using telephone and voice communications. Such fraudulent activities can cause significant damage, especially to elderly people and those unfamiliar with technology. Conventional methods have made it difficult to detect fraud and prevent damage before it occurs. Therefore, there is a need for a system that can detect potentially fraudulent conversations in real time and issue a prompt alert.

[0257] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0258] In this invention, the server includes means for collecting voice, means for converting the collected voice into text data using voice recognition technology, means for analyzing the converted text data using a natural language processing model to detect conversations that may be fraudulent, and means for issuing an alert when a conversation that may be fraudulent is detected. This makes it possible to detect conversations that may be fraudulent in real time and to issue an alert quickly.

[0259] The "means for collecting voice" refers to a device or function for acquiring the user's conversation or voice in real time.

[0260] "Speech recognition technology" is a technology for converting acquired voice data into text data, allowing voice to be treated as text information.

[0261] "Text data" is character information converted using speech recognition technology, and is data in a format that can be analyzed and processed.

[0262] A "natural language processing model" is an artificial intelligence model that analyzes text data and determines whether it is likely to be fraudulent.

[0263] A "potentially fraudulent conversation" is a conversation that, after analysis by a natural language processing model, is determined to contain signs of fraudulent activity.

[0264] An "alert issuing means" is a device or function that issues a warning or notification to a user or a third party when a potentially fraudulent conversation is detected.

[0265] A "dedicated device" is a device specialized for collecting audio, including, for example, a smartphone or smart speaker.

[0266] A "generative AI model" is an artificial intelligence model that is used to analyze text data and determine the likelihood of fraud, and is trained for a specific task.

[0267] The present invention relates to a system that detects potentially fraudulent conversations in real time and issues an alert quickly. A specific embodiment of this system will be described below.

[0268] First, the device is equipped with a voice collection means for capturing the user's conversation in real time. The device is equipped with a microphone and records what the user says. Specific hardware used for this is a smartphone or smart speaker.

[0269] The device then converts the captured voice data into text using voice recognition technology. This process uses voice recognition software such as the Google Speech-to-Text API. Once the voice data is converted into text, it can be handled as text information.

[0270] The device then inputs the converted text data into a natural language processing model, specifically a generative AI model such as OpenAI's GPT-4, to analyze the text data and determine whether it is likely to be fraudulent. The model is trained to detect keywords and contexts related to fraud.

[0271] If the device determines that there is a high possibility of fraud based on the analysis results of the NLP model, it sets a fraud detection flag, which triggers the subsequent alert sending process.

[0272] Finally, the device will issue an alert if a fraud detection flag is raised. The alert can take multiple forms, including a voice message from the device or an automated call to registered contacts (e.g., family members or the police). Specifically, an automated call can be made using the Twilio API.

[0273] As a concrete example, consider the case where a user receives a phone call and is told, "Your bank account has been fraudulently used. Please tell us your account number and PIN to verify."

[0274] Process flow:

[0275] 1. The device picks up the user's conversation using a microphone. For example, the microphone on a smartphone records the user's voice.

[0276] 2. The device uses the Google Speech-to-Text API to convert the speech to text, which reads, "Your bank account has been compromised. Please provide your account number and PIN to verify."

[0277] 3. The device inputs the converted text into GPT-4 to analyze the possibility of fraud. GPT-4 detects keywords such as "bank account," "fraudulent use," "account number," and "PIN number," and determines that there is a high possibility of fraud.

[0278] 4. The device determines that there is a high possibility of fraud and sets a fraud detection flag.

[0279] 5. The device automatically calls the user's family using the Twilio API and notifies them that a potentially fraudulent conversation has been detected.

[0280] Example prompts to input to a generative AI model:

[0281] "When a user calls and says, 'Your bank account has been compromised. Please provide your account number and PIN to verify,' analyze whether this is a potential fraud."

[0282] This system makes it possible to prevent fraudulent acts before they occur. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0283] Step 1:

[0284] The device captures the user's conversation in real time. Specifically, it uses a microphone installed on the device to collect voice data. The input is the user's voice, and the output is digital voice data. This voice data is used in subsequent processing steps.

[0285] Step 2:

[0286] The device converts the collected voice data into text data using voice recognition technology. Specifically, it uses the Google Speech-to-Text API to analyze the voice data and generate corresponding text data. The input is voice data and the output is text data. This text data is used in subsequent natural language processing steps.

[0287] Step 3:

[0288] The device inputs the converted text data into a natural language processing model. Specifically, it uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the text data and determine whether it is likely to be fraud. The input is the text data, and the output is an analysis result indicating the likelihood of fraud. This analysis result is used to raise a fraud detection flag.

[0289] Step 4:

[0290] If the device determines that there is a high possibility of fraud based on the analysis results of the NLP model, it sets a fraud detection flag. The input is the analysis results of the NLP model, and the output is a fraud detection flag. This flag triggers the subsequent alert sending process.

[0291] Step 5:

[0292] If a fraud detection flag is raised, the device will issue an alert. Specifically, this can be a voice message from the device or an automated call to registered contacts (e.g., family members or the police). The input is the fraud detection flag and the output is an alert notification. Specific actions include making an automated call using the Twilio API.

[0293] (Application example 3)

[0294] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0295] In recent years, fraud has been on the rise, particularly over the phone. This has resulted in many people suffering financial losses. Conventional methods have made it difficult to detect potentially fraudulent conversations in real time and respond quickly. Therefore, there is a need for a system that can detect potentially fraudulent conversations in real time, quickly issue alerts, and, if necessary, automatically communicate with registered contacts.

[0296] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0297] In this invention, the server includes means for collecting audio, means for analyzing the collected audio and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, and means for automatically communicating with registered contacts when a conversation that may be fraudulent is detected. This makes it possible to detect conversations that may be fraudulent in real time and respond quickly.

[0298] "Means for collecting audio" refers to devices and software for acquiring audio data.

[0299] "Means for analyzing collected audio and detecting potentially fraudulent conversations" refers to devices or software that analyze acquired audio data and identify content that may be fraudulent.

[0300] An "alert mechanism" is a device or software that issues a warning when it detects a potentially fraudulent conversation.

[0301] "Means for automatically communicating with registered contacts" refers to devices or software that automatically communicate with pre-registered contacts when a potentially fraudulent conversation is detected.

[0302] A "purpose-built terminal" is an electronic device designed for a specific purpose.

[0303] A "generative AI model" is a model generated using artificial intelligence techniques and trained to perform a specific task.

[0304] A system for implementing this invention includes means for collecting audio, means for analyzing the collected audio and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, and means for automatically communicating with registered contacts when a conversation that may be fraudulent is detected. Specific embodiments are described below.

[0305] Hardware and Software Configuration

[0306] The system is implemented using mobile devices such as smartphones. It uses the smartphone's microphone to collect audio. It uses the Python programming language and the SpeechRecognition library to analyze the audio data. It uses a generative AI model to detect potentially fraudulent conversations. It uses the Twilio API to trigger voice messages and automated phone calls to issue alerts.

[0307] Data processing and calculation

[0308] The device first collects voice data through a microphone. The collected voice data is converted to text data using the SpeechRecognition library. The text data is then analyzed using a generative AI model to detect potentially fraudulent keywords and phrases. If a potentially fraudulent conversation is detected, the device uses the Twilio API to alert the user with a voice message and automatically make a call to a registered contact.

[0309] Specific examples

[0310] For example, if a user says, "Please tell me my bank account information," the system will detect the keyword "bank account," issue an alert, and automatically call a registered contact.

[0311] Prompt Sentence Examples

[0312] "Create an application that detects potentially fraudulent conversations and alerts users. Include functionality to capture voice data and trigger alerts if fraudulent keywords are detected."

[0313] In this way, the present invention is able to detect potentially fraudulent conversations in real time and respond quickly.

[0314] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0315] Step 1:

[0316] The device collects voice data using the smartphone's microphone. When the user starts a call, the device continues to capture voice data in real time. The input is the user's voice, and the output is the collected voice data.

[0317] Step 2:

[0318] The device converts the collected voice data into text data using the SpeechRecognition library. To convert voice data into text data, it uses Google's speech recognition API. The input is voice data, and the output is the converted text data.

[0319] Step 3:

[0320] The device analyzes text data using a generative AI model to detect potentially fraudulent keywords and phrases. The generative AI model scans the text data using a pre-trained list of fraudulent keywords. The input is the text data, and the output is a judgment of whether it is potentially fraudulent.

[0321] Step 4:

[0322] If a potentially fraudulent conversation is detected, the device uses the Twilio API to alert the user with a voice message. The input is the result of the potentially fraudulent conversation, and the output is an audio warning to the user.

[0323] Step 5:

[0324] Furthermore, if a potentially fraudulent conversation is detected, the device will automatically call registered contacts using the Twilio API. The input is the result of the potentially fraudulent conversation, and the output is an automated call to the registered contact.

[0325] In this way, the device can detect potentially fraudulent conversations in real time and respond quickly.

[0326] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0327] "Example 1"

[0328] In one embodiment of the present invention, the system includes an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's voice and issues an alert based on the user's emotional state. For example, if the user expresses negative emotions such as anger or fear, the system determines that fraud is a possibility and issues an alert.

[0329] "Example 2"

[0330] In addition, the emotion engine strengthens the detection of potentially fraudulent conversations when the user's emotional state deteriorates above a certain threshold. For example, if the user shows strong anger or fear, the emotion engine will detect that emotional state and strengthen the detection of potentially fraudulent conversations. This allows for more accurate detection of situations in which the user is likely to be scammed.

[0331] "Example 3"

[0332] Furthermore, the emotion engine monitors the user's emotional state in real time and controls the issuance of alerts according to changes in the emotional state. For example, if the user's emotion changes from anger or fear to relief or joy, the emotion engine detects this change in emotional state and stops issuing alerts. This allows the system to provide appropriate alerts according to the user's emotional state.

[0333] The processing flow of each embodiment will be described below.

[0334] "Example 1"

[0335] Step 1: Collect the user's voice using a dedicated device.

[0336] Step 2: The collected voice data is sent to the emotion engine to recognize the user's emotions.

[0337] Step 3: Detect potentially fraudulent conversations based on the emotional state recognized by the emotion engine.

[0338] Step 4: If a potentially fraudulent conversation is detected, issue an alert.

[0339] "Example 2"

[0340] Step 1: Collect the user's voice using a dedicated device.

[0341] Step 2: The collected voice data is sent to the emotion engine to recognize the user's emotions.

[0342] Step 3: Enhance detection of potentially fraudulent conversations when the emotional state recognized by the emotion engine deteriorates above a certain threshold.

[0343] Step 4: If a potentially fraudulent conversation is detected, issue an alert.

[0344] "Example 3"

[0345] Step 1: Collect the user's voice using a dedicated device.

[0346] Step 2: The collected voice data is sent to the emotion engine to recognize the user's emotions.

[0347] Step 3: Monitor the emotional state recognized by the emotion engine in real time and control the issuance of alerts according to changes in the emotional state.

[0348] Step 4: If the user's emotional state changes to a positive state, such as relief or joy, stop issuing alerts.

[0349] Example 1

[0350] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0351] In order to reduce the risk of elderly people falling victim to fraud, it is important to detect signs of fraud early in everyday conversations and take appropriate action. However, conventional systems lack sufficient analysis of voice data and emotion recognition, making it difficult to accurately determine the possibility of fraud. In addition, there is a lack of means to evaluate the user's emotional state, which can lead to delayed response when the user displays negative emotions. This has led to the issue of increasing the risk of elderly people falling victim to fraud.

[0352] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0353] In this invention, the server includes means for collecting voice, means for analyzing the collected voice to detect conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, means for converting voice data into text data, means for recognizing emotions from the text data, and means for evaluating the user's emotional state based on the emotion recognition results. This makes it possible to accurately recognize emotions from voice data and detect the possibility of fraud early. Furthermore, by evaluating the user's emotional state, it is possible to respond quickly when negative emotions are displayed, thereby reducing the risk of elderly people being scammed.

[0354] "Audio collection means" refers to a device or equipment for capturing ambient sounds, particularly a dedicated terminal that includes a highly sensitive microphone.

[0355] "Means for analyzing collected audio to detect potentially fraudulent conversations" refers to algorithms or software that analyze voice data and identify signs of fraud from its content.

[0356] "Means for issuing alerts when potentially fraudulent conversations are detected" refers to systems or features that send notifications to designated contacts when a potential fraudulent conversation is detected.

[0357] "Means for converting voice data into text data" refers to software or algorithms for converting voice data into text information using voice recognition technology.

[0358] "Means for recognizing emotions from text data" refers to emotion analysis algorithms or software that analyze text data and identify user emotions from its content.

[0359] "Means for evaluating the user's emotional state based on the emotion recognition results" refers to a system or function that comprehensively evaluates the user's emotional state based on the results of emotion analysis and takes appropriate measures as necessary.

[0360] This invention is a system for reducing the risk of elderly people falling victim to fraud, and by collecting and analyzing voice and recognizing emotions, it can detect the possibility of fraud early and issue an alert. A specific embodiment of this system will be described below.

[0361] Audio Collection

[0362] The device is installed in a place frequently used by the elderly, such as the living room or bedroom. The device is equipped with a highly sensitive microphone that constantly collects surrounding sounds. For example, the device can capture the elderly's voice saying, "Hello, how was your day?"

[0363] Sending audio data

[0364] The device transmits the collected audio data to the server in real time. The audio data is encrypted and transmitted using a secure communication protocol (e.g., HTTPS). For example, the device encrypts the audio data it captures and transmits it to the server using the HTTPS protocol.

[0365] Analysis of audio data

[0366] The server uses speech recognition software (e.g., Google Cloud Speech-to-Text API) to analyze the received voice data. The server converts the voice data into text. For example, the voice data "Hello, how was your day?" is converted into text data "Hello, how was your day?"

[0367] emotion recognition

[0368] The server inputs the text data into an emotion analysis algorithm (e.g., IBM Watson® Tone Analyzer) to recognize the user's emotion. The emotion analysis algorithm identifies the type of emotion (e.g., anger, fear, joy, etc.) from the text data. For example, the phrase "Hello, how was your day?" is analyzed from the text data, and the emotion is recognized as "neutral."

[0369] Alert sending

[0370] The server evaluates the user's emotional state based on the results of emotion analysis. If the user shows negative emotions such as anger or fear, the server determines that there is a possibility of fraud and issues an alert. The alert is sent to designated contacts (e.g., family members or caregivers) via email or SMS. For example, if the user utters the text data "Help me, I'm scared!", the server recognizes the emotion as "fear" and determines that there is a possibility of fraud. The server then sends an email alert to the family saying, "Mom is scared. Please check on her."

[0371] Prompt Sentence Examples

[0372] An example of a prompt to be input to the generative AI model is as follows:

[0373] "Describe a system that collects audio data from elderly people having conversations in their living rooms and recognizes emotions from the audio. Include the ability to issue an alert if the user is expressing anger or fear."

[0374] Using this prompt, the generative AI model can generate a detailed description of the system.

[0375] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0376] Step 1:

[0377] The device is installed in a place frequently used by the elderly, such as the living room or bedroom. The device uses a highly sensitive microphone to constantly collect surrounding sounds. The input is the surrounding sounds, and the output is the collected voice data. For example, the device captures the elderly person's voice saying, "Hello, how was your day?"

[0378] Step 2:

[0379] The device transmits the collected audio data to the server in real time. The audio data is encrypted and transmitted using a secure communication protocol (e.g., HTTPS). The input is the collected audio data, and the output is the encrypted audio data. For example, the device encrypts the audio data it captures and transmits it to the server using the HTTPS protocol.

[0380] Step 3:

[0381] The server uses speech recognition software (e.g., Google Cloud Speech-to-Text API) to analyze the received voice data. The server converts the voice data into text. The input is encrypted voice data, and the output is text data. For example, the voice data "Hello, how was your day?" is converted into text data "Hello, how was your day?"

[0382] Step 4:

[0383] The server inputs the text data into an emotion analysis algorithm (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. The emotion analysis algorithm identifies the type of emotion (e.g., anger, fear, joy, etc.) from the text data. The input is the text data, and the output is the emotion recognition result. For example, the phrase "Hello, how was your day?" is analyzed from the text data, and the emotion is recognized as "neutral."

[0384] Step 5:

[0385] The server evaluates the user's emotional state based on the results of emotion analysis. If the user shows negative emotions such as anger or fear, the server determines that there is a possibility of fraud and issues an alert. The input is the emotion recognition result, and the output is an alert notification. For example, if the user utters text data such as "Help, I'm scared!", the server recognizes the emotion as "fear" and determines that there is a possibility of fraud. The server then sends an email alert to the family saying, "Mom is scared. Please check on her."

[0386] (Application example 1)

[0387] Next, a description will be given of Application Example 1 of Embodiment Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0388] In order to reduce the risk of elderly people becoming victims of fraud, it is necessary to detect potentially fraudulent conversations in real time and respond quickly. However, conventional systems have issues with not being able to completely prevent fraud because they are unable to collect and analyze voice data properly and are unable to send alerts that take into account changes in emotions.

[0389] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for collecting voice, means for analyzing the collected voice and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, means for recognizing emotions, and means for issuing an alert when the recognized emotions are negative. This reduces the risk of elderly people becoming victims of fraud and enables prompt response.

[0390] "Means for collecting voice" refers to devices or software for collecting the user's voice in real time.

[0391] "Means for analyzing collected audio to detect potentially fraudulent conversations" refers to algorithms or programs that analyze collected audio data and identify potentially fraudulent conversations from their content.

[0392] "Means for issuing an alert when a potentially fraudulent conversation is detected" refers to a device or software that issues a warning to a user or a third party when a potentially fraudulent conversation is detected.

[0393] "Means for recognizing emotions" refers to algorithms or programs that analyze a user's emotions from voice data and recognize a specific emotional state.

[0394] "Means for issuing an alert when the recognized emotion is negative" refers to a device or software for issuing a warning to a user or a third party when the recognized emotion is a negative state such as anger or fear.

[0395] "Specialized Device" means a hardware device designed and manufactured for a specific purpose.

[0396] A "generative AI model" is a model generated using artificial intelligence techniques and refers to an algorithm trained to perform a specific task.

[0397] This invention is a system for reducing the risk of elderly people becoming victims of fraud, and it has the ability to collect voice, recognize emotions, and detect potentially fraudulent conversations. The system is realized using a dedicated device, an emotion recognition engine, and a generative AI model.

[0398] System configuration

[0399] The system consists of the following main components:

[0400] 1. Dedicated terminal: A hardware device that is installed in a location frequently used by elderly people to collect their voice. This terminal has a built-in microphone and collects voice in real time.

[0401] 2. Emotion Recognition Engine: Software that analyzes user emotions from collected voice data and recognizes specific emotional states. This engine takes voice data as input and outputs an emotional state.

[0402] 3. Generative AI model: An algorithm that analyzes collected voice data and detects potentially fraudulent conversations. This model takes voice data as input and determines whether it is likely to be fraudulent.

[0403] 4. Alerting system: Software that warns users and third parties when potentially fraudulent conversations or negative sentiment are detected. Alerts are sent via email and smartphone notifications.

[0404] Program processing explanation

[0405] The server collects voice data from a dedicated device and analyzes it using an emotion recognition engine and generative AI model. Specifically, the following process is carried out:

[0406] 1. Voice collection: A dedicated device collects the voice of the elderly person in real time. The collected voice data is sent to a server.

[0407] 2. Emotion Recognition: The server uses an emotion recognition engine to analyze the user's emotions from the voice data. For example, if the user expresses anger or fear, the emotional state is recognized.

[0408] 3. Fraud Detection: The server analyzes the audio data using a generative AI model to detect potentially fraudulent conversations. If fraud is detected, the information is sent to an alert system.

[0409] 4. Alerting: When potentially fraudulent conversations or negative sentiment are detected, the alerting system will warn users or third parties via email or smartphone notifications.

[0410] Specific examples

[0411] For example, if an elderly person is talking to a scammer on the phone, and emotions such as "anger" or "fear" are detected in the conversation, an alert email will be sent to the family stating, "An elderly person is showing signs of anger. There is a possibility of fraud."

[0412] Prompt Sentence Examples

[0413] "Develop a system that collects voices from elderly people and recognizes their emotions. If emotions such as anger or fear are detected in the voice, the application will send an alert email to their family members. The libraries used are speech_recognition, emotion_recognition, and smtplib."

[0414] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0415] Step 1:

[0416] A dedicated terminal collects the elderly person's voice in real time. The terminal's microphone captures the voice data and sends it to a server. The input is the elderly person's voice, and the output is the voice data sent to the server.

[0417] Step 2:

[0418] The server inputs the received voice data into an emotion recognition engine, which analyzes the voice data and identifies the user's emotional state. The input is the voice data, and the output is the recognized emotional state (e.g., anger, fear).

[0419] Step 3:

[0420] The server evaluates the emotional state obtained from the emotion recognition engine, and if a negative emotion (e.g., anger, fear) is detected, it sends the information to the alerting system. The input is the recognized emotional state, and the output is the emotional information sent to the alerting system.

[0421] Step 4:

[0422] The server inputs the voice data into a generative AI model, which analyzes the voice data and detects potentially fraudulent conversations. The input is the voice data, and the output is a judgment result on whether or not there is a possibility of fraud.

[0423] Step 5:

[0424] The server evaluates the judgment results obtained from the generative AI model, and if it determines that there is a possibility of fraud, it sends that information to the alert system. The input is the judgment result of whether there is a possibility of fraud, and the output is the fraud information sent to the alert system.

[0425] Step 6:

[0426] The alert system issues warnings to users and third parties based on emotional information and fraud information received from the server. The warnings are sent via email or smartphone notifications. The input is emotional information and fraud information, and the output is a warning message sent to users and third parties.

[0427] Example 2

[0428] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0429] While conventional voice analysis systems can detect potentially fraudulent conversations, they have the problem of making it difficult to accurately assess the risk of fraud because they do not take the user's emotional state into account. Furthermore, they are unable to adequately detect situations in which the likelihood of fraud increases when the user is expressing strong emotions, which makes it difficult to adequately ensure the user's safety.

[0430] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0431] In this invention, the server includes means for collecting audio, means for analyzing the collected audio to detect potentially fraudulent conversations, means for issuing an alert when potentially fraudulent conversations are detected, means for detecting the user's emotional state, and means for enhancing the detection of potentially fraudulent conversations when the user's emotional state deteriorates to a certain threshold or greater, thereby enabling accurate determination of fraud risk taking the user's emotional state into account.

[0432] A "means for collecting voice" is a device or software for acquiring voice data provided by a user in digital form and transmitting it to a server.

[0433] "Means for analyzing collected audio to detect potentially fraudulent conversations" refers to technology that converts audio data into text and analyzes that text to identify potentially fraudulent phrases and patterns.

[0434] The "means for issuing an alert when a potentially fraudulent conversation is detected" refers to a device or software for issuing a warning to a user when a potentially fraudulent conversation is detected.

[0435] "Means for detecting the user's emotional state" refers to technology for analyzing the user's voice and facial expression data to evaluate their emotional state.

[0436] "Means for strengthening the detection of potentially fraudulent conversations when a user's emotional state deteriorates above a certain threshold" refers to technology that more strictly detects potentially fraudulent conversations when a user is expressing strong emotions such as anger or fear.

[0437] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to analyze text data and identify specific patterns and phrases.

[0438] A "prompt sentence" is text data that is input into a generative AI model and is a sentence that specifically indicates the content of the object to be analyzed.

[0439] The present invention relates to a system for collecting voice data and detecting potentially fraudulent conversations. A specific embodiment of this system will be described below.

[0440] Audio data collection

[0441] The user provides voice data. The device collects this voice using a built-in microphone and converts it into a digital format. The device then transmits the collected voice data to a server.

[0442] Converting audio data to text

[0443] The server receives the voice data sent from the device. The server calls the Google Cloud Speech-to-Text API and converts the voice data into text. For example, the text generated is "Hello, how are you doing lately?"

[0444] Detecting potential fraud

[0445] The server inputs the generated text into a generative AI model (e.g., OpenAI's GPT-4). The generative AI model uses natural language processing techniques to analyze whether the text is likely to be fraudulent. For example, if the text contains the phrase "please send me money," it will determine that it is likely to be fraudulent.

[0446] Emotional state detection

[0447] The device collects the user's voice and facial expression data and sends it to a server, which then uses Affectiva's emotion recognition API to analyze the user's emotional state. For example, if the user is expressing strong anger or fear, the server can detect that emotional state.

[0448] Enhanced fraud detection

[0449] The server will strengthen its detection of potentially fraudulent conversations if the user's emotional state deteriorates above a certain threshold. For example, if the user expresses strong fear when asked to "send money," it will determine that the conversation is likely to be fraudulent.

[0450] Notification of results

[0451] If the server determines that there is a possibility of fraud, it notifies the terminal of the result. The terminal displays a warning to the user, saying, "There is a possibility of fraud. Please be careful."

[0452] Specific examples

[0453] The user says, "I want you to send me money." The device collects this audio and sends it to the server. The server converts the audio into text and inputs it into a generative AI model. The generative AI model determines that there is a possibility of fraud. At the same time, the device collects the user's facial expression data and sends it to the server. The server uses Affectiva's emotion recognition API to detect that the user is showing strong fear. The server determines that there is a high possibility of fraud and notifies the device to display a warning. The device displays a message to the user saying, "This may be a scam. Please be careful."

[0454] Prompt Sentence Examples

[0455] "Please help us determine if the following text is a potential scam: 'I want you to send me money'"

[0456] This system is expected to protect users from the risk of fraud.

[0457] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0458] Step 1:

[0459] The user provides voice data. The device uses the built-in microphone to collect this voice and convert it into digital form. The input is the user's voice data, and the output is digital voice data. In concrete terms, the device captures voice through the microphone and performs digital signal processing.

[0460] Step 2:

[0461] The terminal transmits the collected digital voice data to the server. The input is the digital voice data, and the output is the voice data transmitted to the server. Specifically, the terminal uploads the voice data to the server via a network.

[0462] Step 3:

[0463] The server receives the voice data sent from the terminal. The input is the voice data sent from the terminal, and the output is the voice data stored in the server. In concrete terms, the server receives the voice data via the network and stores it in a database.

[0464] Step 4:

[0465] The server calls the Google Cloud Speech-to-Text API to convert the voice data to text. The input is the voice data stored on the server, and the output is text data. Specifically, the server calls the API and sends a request to convert the voice data to text.

[0466] Step 5:

[0467] The server inputs the generated text data into a generative AI model (for example, OpenAI's GPT-4) and analyzes whether there is a possibility of fraud. The input is text data, and the output is the analysis result regarding the possibility of fraud. Specifically, the server inputs the text data into the generative AI model and obtains the analysis result.

[0468] Step 6:

[0469] The device collects the user's voice and facial expression data and sends it to the server. The input is the user's voice and facial expression data, and the output is the emotion data sent to the server. Specifically, the device uses a camera and microphone to capture the user's facial expressions and voice, converts them into digital format, and sends them to the server.

[0470] Step 7:

[0471] The server uses Affectiva's emotion recognition API to analyze the user's emotional state. The input is the emotional data sent to the server, and the output is the analysis result of the user's emotional state. Specifically, the server calls the emotion recognition API and sends a request to analyze the emotional data.

[0472] Step 8:

[0473] The server enhances detection of potentially fraudulent conversations when the user's emotional state deteriorates above a certain threshold. The input is the analysis result of the user's emotional state, and the output is the enhanced fraud detection result. Specifically, the server evaluates the analysis result of the emotional state and adjusts the threshold of the fraud detection algorithm.

[0474] Step 9:

[0475] If the server determines that there is a possibility of fraud, it notifies the terminal of the result. The input is the enhanced fraud detection result, and the output is a warning message sent to the terminal. Specifically, the server generates a warning message and sends it to the terminal via the network.

[0476] Step 10:

[0477] The terminal displays a warning to the user, informing them that "This may be a scam. Please be careful." The input is the warning message sent from the server, and the output is the warning message displayed to the user. Specifically, the terminal displays the warning message on the screen and notifies the user.

[0478] (Application example 2)

[0479] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0480] In recent years, telephone and internet-based fraud has been on the rise, with many people falling victim to it. Elderly people and those unfamiliar with technology are particularly vulnerable targets, and fraud methods are becoming more sophisticated, making it difficult to adequately counter these attacks using conventional methods. Furthermore, it is necessary to not only detect potentially fraudulent conversations, but also to more accurately assess the likelihood of fraud by taking into account the user's emotional state. This will reduce the risk of users falling victim to fraud and provide safer communications.

[0481] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0482] In this invention, the server includes means for collecting audio, means for analyzing the collected audio to detect potentially fraudulent conversations, means for issuing an alert when potentially fraudulent conversations are detected, means for detecting a user's emotional state, and means for enhancing detection of potentially fraudulent conversations when the user's emotional state deteriorates to a certain threshold or greater. This makes it possible to detect potentially fraudulent conversations in real time and issue a warning taking the user's emotional state into consideration.

[0483] The "means for collecting voice" is a device or software for capturing the user's voice in real time.

[0484] "Means for analyzing collected audio to detect potentially fraudulent conversations" refers to technology that analyzes acquired audio data and detects phrases or patterns that may be fraudulent.

[0485] The "means for issuing an alert when a potentially fraudulent conversation is detected" is a device or software for issuing a warning to a user when a potentially fraudulent conversation is detected.

[0486] The "means for detecting the user's emotional state" is a technology for analyzing and detecting the user's emotional state from their voice or other data.

[0487] "Means for strengthening detection of potentially fraudulent conversations when a user's emotional state deteriorates beyond a certain threshold" refers to technology for more rigorously detecting potentially fraudulent conversations when a user's emotional state deteriorates.

[0488] A system for implementing the present invention includes a set of means for collecting audio, detecting potentially fraudulent conversations, and issuing a warning to the user. A specific embodiment of this system is described below.

[0489] First, the smartphone's microphone is used to collect audio. The smartphone monitors the user's calls in real time and collects audio data. The collected audio data is converted into text using speech recognition software (e.g., the speech_recognition library).

[0490] Next, a chat generation AI model (for example, a model using the transformers library) is used to analyze the collected audio and detect potentially fraudulent conversations. This model analyzes the converted text and detects phrases and patterns that may be fraudulent. For example, if a phrase such as "please send money" is included, it is determined to be fraudulent.

[0491] Furthermore, emotion recognition software (e.g., emotion_recognition library) is used as a means of detecting the user's emotional state. This software analyzes and detects the user's emotional state from voice and text data. If the user shows strong anger or fear, it is determined that the user's emotional state is deteriorating.

[0492] The system will integrate emotion recognition software with chat generation AI models as a means of strengthening detection of potentially fraudulent conversations when a user's emotional state deteriorates above a certain threshold. When the user's emotional state deteriorates, the system will more strictly detect potentially fraudulent conversations.

[0493] The system uses the smartphone's notification function to issue an alert when a conversation that may be fraudulent is detected. When a conversation that may be fraudulent is detected, a warning is issued to the user to warn them.

[0494] As a concrete example, the following prompt sentence is input into the generative AI model:

[0495] User: Hello, how are you doing?

[0496] Scammer: Hello, I'm actually in a bit of a pinch and I'd like you to send me some money.

[0497] When the user enters this prompt, the chat generation AI model will determine that it is a potential scam and issue a warning to the user. If the user expresses strong anger or fear, emotion recognition software will detect that emotional state and reinforce the warning.

[0498] In this way, potentially fraudulent conversations can be detected in real time and alerted taking into account the user's emotional state.

[0499] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0500] Step 1:

[0501] The device collects the user's voice. Specifically, it uses the smartphone's microphone to capture voice data during a call in real time. The input is the user's voice, and the output is voice data.

[0502] Step 2:

[0503] The device converts the collected voice data into text. Specifically, it uses the speech_recognition library to perform voice recognition. The input is voice data, and the output is text data.

[0504] Step 3:

[0505] The server analyzes the converted text data and detects potentially fraudulent conversations. Specifically, it uses the transformers library to input text data into a chat generation AI model to detect potentially fraudulent phrases and patterns. The input is text data, and the output is a judgment result on whether or not there is a possibility of fraud.

[0506] Step 4:

[0507] The server detects the user's emotional state. Specifically, it analyzes the emotional state from text data using the emotion_recognition library. The input is text data, and the output is the analysis result of the emotional state.

[0508] Step 5:

[0509] The server strengthens its detection of potentially fraudulent conversations when the user's emotional state deteriorates above a certain threshold. Specifically, when the emotional state is strong, such as "anger" or "fear," it more strictly detects potentially fraudulent phrases and patterns. The input is the analysis result of the emotional state, and the output is the judgment result of the enhanced fraud detection.

[0510] Step 6:

[0511] The device issues an alert when a potentially fraudulent conversation is detected. Specifically, it displays a warning to the user using the smartphone's notification function. The input is the fraud detection result, and the output is a warning notification to the user.

[0512] Example 3

[0513] Next, a description will be given of a third embodiment of the third embodiment. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0514] Conventional fraud detection systems focus on detecting potentially fraudulent conversations, but because they do not take into account the user's emotional state, they may issue alerts at the wrong time. Furthermore, they lack the ability to appropriately control alerts when the user's emotional state changes, which can cause unnecessary stress to the user. This creates a problem of impairing the user experience.

[0515] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0516] In this invention, the server includes means for collecting voice, means for analyzing the collected voice and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, means for monitoring the user's emotional state in real time, and means for controlling the issuance of the alert in response to changes in the user's emotional state. This makes it possible not only to detect conversations that may be fraudulent, but also to provide appropriate alerts that correspond to the user's emotional state.

[0517] "Voice collection means" refers to a device or function for acquiring the user's voice and storing it as digital data.

[0518] "Means for analyzing collected audio and detecting potentially fraudulent conversations" refers to devices or functions that convert acquired audio data into text data and analyze that text data to detect keywords or phrases that may be fraudulent.

[0519] "Means for issuing an alert when a potentially fraudulent conversation is detected" means a device or function for issuing a warning to a user when a potentially fraudulent conversation is detected, and includes forms such as a voice message or automated phone call.

[0520] The "means for monitoring the user's emotional state in real time" refers to a device or function for analyzing the user's voice data and grasping the user's emotional state in real time.

[0521] The "means for controlling the generation of an alert in response to a change in the user's emotional state" refers to a device or function for appropriately controlling the generation of an alert in response to a change in the user's emotional state.

[0522] MODE FOR CARRYING OUT THE INVENTION

[0523] The present invention is a system for detecting potentially fraudulent conversations and providing an appropriate alert to the user. A specific embodiment of this system will be described below.

[0524] Hardware and software used

[0525] The system uses a microphone to receive voice input, a speaker or phone-enabled device to generate alerts, and software including a speech recognition engine, emotion engine, and alert management system. The software uses Google Cloud Speech-to-Text API for speech recognition and IBM Watson Sentiment Analysis API for emotion analysis.

[0526] Data processing and calculation

[0527] 1. Acquiring voice input:

[0528] When the user speaks, the terminal acquires voice data through a microphone.

[0529] What it does: The device's microphone captures the user's voice and stores it as digital audio data.

[0530] 2. Speech Recognition:

[0531] The device sends the acquired voice data to the Google Cloud Speech-to-Text API and converts it into text data.

[0532] Specific operation: The device uploads the voice data to the cloud and receives the text data returned by the API.

[0533] 3. Fraud detection:

[0534] The device then analyzes the converted text data to detect potentially fraudulent keywords and phrases.

[0535] What it does: The device feeds the text data into an internal fraud detection algorithm to detect risky phrases like "Can you tell me your bank account number?"

[0536] 4. Emotional state monitoring:

[0537] The device sends the user's voice data to IBM Watson's emotion analysis API to monitor the user's emotional state in real time.

[0538] How it works: The device uploads voice data to the cloud and receives emotional state data returned by the API, determining whether the user is feeling angry or scared, for example.

[0539] 5. Alert occurs:

[0540] The device will issue an alert if it detects a potentially fraudulent conversation, which can take the form of a voice message or automated phone call.

[0541] Specific behavior: The device will issue a voice warning saying "This may be a scam. Please be careful." If necessary, it will also automatically call registered contacts (for example, family members or the police).

[0542] The emotion engine will stop issuing alerts if the user's emotional state changes to relief or joy.

[0543] Specific behavior: The emotion engine monitors the user's emotional state and stops issuing alerts if emotions such as relief or joy are detected.

[0544] Specific examples

[0545] 1. Detecting potentially fraudulent conversations:

[0546] Scenario: A user calls and says, "What's your bank account number?"

[0547] process:

[0548] The device captures the audio and converts it into text using the Google Cloud Speech-to-Text API.

[0549] The device analyzes the text data and detects the phrase "Please tell me your bank account number."

[0550] The device will issue a voice warning saying, "This may be a scam. Please be careful."

[0551] 2. Controlling alerts based on changes in emotional state:

[0552] Scenario: The user starts out feeling angry or scared, but as the conversation progresses, they begin to feel relieved or happy.

[0553] process:

[0554] The device sends voice data to IBM Watson's sentiment analysis API to monitor the user's emotional state.

[0555] The emotion engine will stop issuing alerts if the user's emotional state changes to relief or joy.

[0556] Prompt Sentence Examples

[0557] Prompt: "Describe a system that detects potentially fraudulent conversations and alerts the user. Include the names of specific hardware and software, and explain how the data is processed and calculated. Include examples."

[0558] In this way, an embodiment of a system that detects conversations that may be fraudulent and provides an appropriate alert to the user will be described. The flow of the identification process in Example 3 will be described with reference to FIG.

[0559] Program processing flow

[0560] Step 1: Getting voice input

[0561] When the user speaks, the terminal acquires voice data through a microphone.

[0562] Input: User's voice

[0563] What it does: The device's microphone captures the user's voice and stores it as digital audio data.

[0564] Output: Digital audio data

[0565] Step 2: Voice Recognition

[0566] The terminal transmits the acquired voice data to a voice recognition engine and converts it into text data.

[0567] Input: Digital audio data

[0568] Specific operation: The device uploads voice data to the cloud and receives text data returned by a speech recognition engine (e.g., Google Cloud Speech-to-Text API).

[0569] Output: Text data

[0570] Step 3: Fraud detection

[0571] The device then analyzes the converted text data to detect potentially fraudulent keywords and phrases.

[0572] Input: Text data

[0573] What it does: The device feeds the text data into an internal fraud detection algorithm to detect risky phrases like "Can you tell me your bank account number?"

[0574] Output: Potentially fraudulent phrase detection results

[0575] Step 4: Monitor your emotional state

[0576] The terminal transmits the user's voice data to an emotion analysis engine to monitor the user's emotional state in real time.

[0577] Input: Digital audio data

[0578] Specific operation: The device uploads voice data to the cloud and receives emotional state data returned by a sentiment analysis engine (e.g., IBM Watson sentiment analysis API), determining, for example, whether the user is feeling angry or scared.

[0579] Output: Emotional state data

[0580] Step 5: Raise an alert

[0581] The terminal will issue an alert if a potentially fraudulent conversation is detected.

[0582] Input: Possible fraud phrase detection results, emotional state data

[0583] Specific behavior: The device will issue a voice warning saying "This may be a scam. Please be careful." If necessary, it will also automatically call registered contacts (for example, family members or the police).

[0584] Output: Alert generation (voice message, automated phone call)

[0585] Step 6: Controlling alerts due to changes in emotional state

[0586] The emotion engine will stop issuing alerts if the user's emotional state changes to relief or joy.

[0587] Input: Emotional state data

[0588] Specific behavior: The emotion engine monitors the user's emotional state and stops issuing alerts if emotions such as relief or joy are detected.

[0589] Output: Alert stopped

[0590] In this way, the process of the system that detects potentially fraudulent conversations and provides appropriate alerts to the user will be described in detail.

[0591] (Application example 3)

[0592] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0593] In recent years, fraud has become more sophisticated, especially with the rise of voice-based fraud. This has increased the risk of many people becoming victims of fraud. Furthermore, while people are becoming more vigilant against fraud, users' emotional states are often unstable. While conventional systems can detect potentially fraudulent conversations, they have the problem of being unable to respond appropriately based on the user's emotional state.

[0594] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0595] In this invention, the server includes means for collecting voice, means for analyzing the collected voice to detect conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, and means for monitoring the user's emotional state in real time and controlling the issuance of the alert in response to changes in the emotional state, thereby making it possible to prevent fraudulent acts and provide appropriate alerts in accordance with the user's emotional state.

[0596] The "means for collecting voice" refers to a device or software for acquiring the user's voice in real time.

[0597] "Means for analyzing collected audio to detect potentially fraudulent conversations" refers to algorithms or generative AI models that analyze acquired audio data and identify potentially fraudulent phrases or patterns.

[0598] "Means for issuing alerts when potentially fraudulent conversations are detected" refers to a voice message or automated phone call function that issues a warning to the user or registered contacts when a potential fraud is detected.

[0599] "Means for monitoring the user's emotional state in real time and controlling the issuance of alerts in response to changes in the emotional state" refers to an emotion engine that analyzes the user's tone of voice and choice of words, detects changes in emotion, and appropriately adjusts the issuance of alerts.

[0600] A "specialized terminal" is a hardware device designed to perform a specific function.

[0601] A "generative AI model" is an artificial intelligence model that is trained to perform a specific task using natural language processing and machine learning techniques.

[0602] A system for implementing this invention includes means for collecting audio, means for analyzing the collected audio to detect potentially fraudulent conversations, means for issuing an alert when potentially fraudulent conversations are detected, and means for monitoring a user's emotional state in real time and controlling the issuance of alerts in response to changes in the emotional state.

[0603] System configuration

[0604] 1. Means of collecting audio:

[0605] The smartphone's microphone is used to capture the user's voice in real time.

[0606] The audio data is converted to text using a speech recognition library (e.g., speech_recognition library).

[0607] 2. Methods for analyzing collected audio to detect potentially fraudulent conversations:

[0608] Generative AI models (e.g., fraud detection models using the transformers library) are used to analyze text data and identify potentially fraudulent phrases and patterns.

[0609] If possible fraud is detected, the relevant text data is flagged.

[0610] 3. How to alert you to potentially fraudulent conversations:

[0611] When potential fraud is detected, alert the user or their registered contacts via voice message (e.g., using the pyttsx3 library) or automated phone call (e.g., using the smtplib library).

[0612] The alert is a message informing you of a possible scam.

[0613] 4. A means for monitoring the user's emotional state in real time and controlling the generation of alerts according to changes in the emotional state:

[0614] Using an emotion engine (e.g., an emotion analysis model using the transformers library), we analyze the user's tone of voice and vocabulary to monitor emotional changes in real time.

[0615] If the user's emotions change from anger or fear to relief or joy, the alerts will stop being generated.

[0616] Specific examples

[0617] If a user is talking to a scammer on the phone and a possible scam is detected, a voice message will be played saying, "This may be a scam. Please be careful," and a warning email will be automatically sent to family members.

[0618] If the user is feeling angry or scared, the emotion engine will detect that and alert them as well.

[0619] Prompt Sentence Examples

[0620] "Could this conversation be a scam?"

[0621] "Analyze the user's emotional state from this text."

[0622] In this way, the fraud prevention alert system can monitor conversations and emotions in real time and provide appropriate alerts to keep users safe.

[0623] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0624] Step 1:

[0625] The device collects the user's voice in real time using the smartphone's microphone. The input is the user's voice, and the output is voice data. This voice data is converted into text using a speech recognition library (e.g., speech_recognition library).

[0626] Step 2:

[0627] The server inputs the collected text data into a generative AI model (e.g., a fraud detection model using the transformers library) to analyze potentially fraudulent phrases and patterns. The input is text data, and the output is a judgment result on whether or not there is a possibility of fraud. If there is a possibility of fraud, the corresponding text data is flagged.

[0628] Step 3:

[0629] If the server determines that there is a possibility of fraud, it generates a voice message (using the pyttsx3 library, for example) and issues a warning to the user through the terminal. The input is the result of the determination that there is a possibility of fraud, and the output is the warning voice message.

[0630] Step 4:

[0631] If the server determines that there is a possibility of fraud, it automatically sends a warning email to registered contacts (e.g., family members or the police). The input is the result of the determination that there is a possibility of fraud, and the output is the warning email. The smtplib library is used to send the email.

[0632] Step 5:

[0633] The server analyzes the user's tone of voice and phrasing using an emotion engine (e.g., an emotion analysis model using the transformers library) to monitor the user's emotional state in real time. The input is text data, and the output is a judgment of the user's emotional state.

[0634] Step 6:

[0635] The server stops issuing alerts when the user's emotional state changes from anger or fear to relief or joy. The input is the result of the emotional state determination, and the output is an instruction to stop issuing alerts.

[0636] In this way, the fraud prevention alert system can collect the user's voice, analyze it for possible fraud, issue warnings as needed, and control the alerts according to the user's emotional state.

[0637] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0638] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0639] Another example of generative AI is Gemini (registered trademark) (Internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[0640] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0641] [Second embodiment]

[0642] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0643] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0644] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0645] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0646] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0647] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0648] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0649] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0650] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0651] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0652] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0653] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[0654] "Example 1"

[0655] In one embodiment of the present invention, voice is collected using a dedicated terminal. This terminal is installed in a place frequently used by the elderly, such as a living room or bedroom. The terminal constantly monitors the elderly's conversations and collects their voice.

[0656] "Example 2"

[0657] The collected audio is analyzed using chat generation AI. The chat generation AI uses natural language processing technology to convert the audio into text and determine whether the text contains conversations that could be fraudulent. For example, if the text contains a phrase such as "I want you to send me money," it is determined to be a potential scam.

[0658] "Example 3"

[0659] If the device detects a potentially fraudulent conversation, it will issue an alert. The alert can be issued in various forms, such as a voice message from the device or an automated call to registered contacts (e.g., family members or the police). This makes it possible to prevent fraud before it occurs.

[0660] The processing flow of each embodiment will be described below.

[0661] "Example 1"

[0662] Step 1: Install a dedicated device in a location frequently used by the elderly, such as the living room or bedroom.

[0663] Step 2: The device constantly monitors the elderly person's conversations and collects their audio.

[0664] "Example 2"

[0665] Step 1: The collected audio is analyzed using chat generation AI.

[0666] Step 2: The chat generation AI uses natural language processing technology to convert speech into text.

[0667] Step 3: Determine if the text contains potentially fraudulent conversations. For example, if it contains a phrase like "please send me money," it is deemed potentially fraudulent.

[0668] "Example 3"

[0669] Step 1: If the device detects a potentially fraudulent conversation, it will issue an alert.

[0670] Step 2: The alert can take various forms, such as a voice message from the device or an automated call to registered contacts (e.g., family members or the police).

[0671] Example 1

[0672] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0673] There is a need to reduce the risk of elderly people becoming victims of fraud and improve their safety in their daily lives. However, conventional systems do not adequately collect and analyze voice data, making it difficult to detect potentially fraudulent conversations in real time. Furthermore, there is a lack of a way for users to easily check the analysis results. This has prevented the realization of effective measures to ensure the safety of elderly people.

[0674] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0675] In this invention, the server includes means for collecting voice, means for analyzing the collected voice and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, means for transmitting the collected voice data to the server, means for the server to analyze the voice data, means for saving the analysis results, and means for the user to check the analysis results. This makes it possible to monitor elderly people's conversations in real time, quickly detect conversations that may be fraudulent, and allow the user to easily check the analysis results.

[0676] The "means for collecting voice" is a dedicated terminal equipped with a highly sensitive microphone, which is a device that constantly monitors the elderly person's conversations and collects voice data.

[0677] The "means of analyzing collected audio to detect potentially fraudulent conversations" refers to a system that uses a generative AI model to convert audio data into text, analyzes the content of that text, and detects potentially fraudulent conversations.

[0678] The "means for issuing an alert when a potentially fraudulent conversation is detected" is a notification system for issuing a warning to the user when a potentially fraudulent conversation is detected.

[0679] The "means for transmitting collected voice data to a server" refers to a function in which the terminal connects to a server via the Internet and encrypts and transmits the collected voice data.

[0680] "Means by which the server analyzes voice data" refers to the process by which the server converts the voice data it receives into text using a generative AI model and analyzes the content.

[0681] The "means for saving the analysis results" is a function that allows the server to save the analysis results in a database, allowing users to easily search for the information they need later.

[0682] The "means for users to check the analysis results" refers to a system that allows users to access the server through a dedicated web interface or mobile app and view the analysis results.

[0683] MODE FOR CARRYING OUT THE INVENTION

[0684] This invention is a system for monitoring elderly people's conversations and detecting potentially fraudulent conversations. The system includes a dedicated terminal, a server, a generative AI model, and a user interface.

[0685] 1. Hardware and Software Configuration

[0686] Terminal

[0687] The device is equipped with a highly sensitive microphone and is installed in places frequently used by elderly people, such as living rooms and bedrooms, and has the function of temporarily storing collected voice data in local storage.

[0688] server

[0689] The server receives voice data from the device via the internet, has a generative AI model (e.g., OpenAI's GPT-4) installed on it, converts the voice data into text, and analyzes its content.

[0690] Generative AI Models

[0691] The generative AI model includes speech recognition technology to convert voice data into text and natural language processing technology to analyze the content of the text, thereby detecting potentially fraudulent conversations.

[0692] User Interface

[0693] Users can access the server and check the analysis results through a dedicated web interface or mobile app. The interface is designed to be easy for users to use.

[0694] 2. Data processing and calculation

[0695] Audio data collection

[0696] The device constantly monitors the elderly person's conversations and collects voice data. For example, if an elderly person is talking to a friend, such as "The weather is nice today," the voice is collected in real time.

[0697] Sending audio data

[0698] The collected voice data is sent from the device to the server at regular intervals (for example, every hour). The data is encrypted before being sent.

[0699] Analysis of audio data

[0700] The server converts the received voice data into text using a generative AI model. This conversion is performed using speech recognition technology. The generative AI model then analyzes the content of the text and extracts important information and unusual patterns. For example, the text "The weather is nice today" is generated and its content is analyzed.

[0701] Saving analysis results

[0702] The server stores the analysis results in a database, including the original audio data, the converted text, and the analysis results, allowing users to easily search for the information they need later.

[0703] Checking the analysis results

[0704] Users can access the server through a dedicated web interface or mobile app and check the analysis results. For example, if a family member wants to check the content of an elderly person's recent conversations, they can log in to the web interface and view the analysis result, such as "The weather is nice today."

[0705] 3. Examples and prompts

[0706] Specific examples

[0707] An elderly person is having a conversation with a friend in the living room, and the content of this conversation is collected by a device and analyzed using a generative AI model.

[0708] Prompt Sentence Examples

[0709] "Collect audio data of an elderly person having a conversation with a friend in their living room, and use a generative AI model to convert the conversation into text."

[0710] The above is an embodiment of the present invention.

[0711] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0712] Step 1:

[0713] The device collects the audio.

[0714] Input: Elderly people's conversation

[0715] How it works: A highly sensitive microphone installed in the device collects the elderly person's conversation in real time. For example, if an elderly person says to a friend, "The weather is nice today," the voice is collected.

[0716] Output: Collected audio data

[0717] Step 2:

[0718] The device stores the audio data in local storage.

[0719] Input: Collected audio data

[0720] Specific operation: The collected voice data is temporarily stored in the local storage of the device.

[0721] Output: Audio data saved in local storage

[0722] Step 3:

[0723] The terminal transmits the voice data to the server.

[0724] Input: Audio data stored in local storage

[0725] Specific operation: At regular intervals (for example, every hour), the device connects to a server via the Internet and encrypts and transmits the collected voice data.

[0726] Output: Audio data sent to the server

[0727] Step 4:

[0728] The server analyzes the audio data.

[0729] Input: Audio data sent to the server

[0730] How it works: The server converts the received voice data into text using a generative AI model (e.g., OpenAI's GPT-4). This conversion uses speech recognition technology. The generative AI model then analyzes the content of the text to extract important information and unusual patterns. For example, the text "The weather is nice today" is generated, and its content is analyzed.

[0731] Output: Parsed text data and analysis results

[0732] Step 5:

[0733] The server stores the analysis results.

[0734] Input: Parsed text data and analysis results

[0735] Specific operation: The server saves the analysis results in a database. The saved data includes the original voice data, the converted text, and the analysis results, allowing users to easily search for the information they need later.

[0736] Output: Analysis results stored in a database

[0737] Step 6:

[0738] The user checks the analysis results.

[0739] Input: Analysis results stored in the database

[0740] Specific operation: Users access the server through a dedicated web interface or mobile app to check the analysis results. For example, if a family member wants to check the content of an elderly person's recent conversations, they can log in to the web interface and view the analysis result, such as "The weather is nice today."

[0741] Output: Analysis results confirmed by the user

[0742] (Application example 1)

[0743] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0744] In the living spaces of elderly people, there is a need for a system that can quickly detect abnormal voice patterns (such as cries for help or the sound of someone falling) and take appropriate action. However, conventional systems have low accuracy in analyzing voice data, and there is a risk that abnormalities will be overlooked. In addition, there is a lack of means to quickly issue an alert when an abnormality is detected, which can delay emergency response. To solve these issues, a system that can perform highly accurate voice analysis and issue alerts quickly is needed.

[0745] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0746] In this invention, the server includes means for collecting voice, means for analyzing the collected voice and detecting abnormal voice patterns, and means for issuing an alert when an abnormal voice pattern is detected. This makes it possible to quickly detect abnormal voice patterns in the elderly person's living space and take appropriate measures.

[0747] The "means for collecting audio" is a dedicated terminal that is installed in the elderly person's living space and is used to constantly collect surrounding audio.

[0748] "Means for analyzing collected voice data and detecting abnormal voice patterns" refers to a function that analyzes collected voice data using a generative AI model to detect abnormal voice patterns such as calls for help or the sound of a fall.

[0749] "Means for issuing an alert when an abnormal voice pattern is detected" is a function that sends an alert to a device such as a smartphone when an abnormal voice pattern is detected, prompting a prompt response.

[0750] A "generative AI model" is an artificial intelligence model used to analyze voice data and detect anomalous voice patterns with high accuracy.

[0751] A "dedicated terminal" is a specialized device that is installed in the elderly person's living space and is used to collect voice.

[0752] The system for implementing this invention consists of a dedicated terminal installed in the elderly person's living space, a voice analysis system using a generative AI model, and a smartphone application that issues an alert when an abnormality is detected.

[0753] Hardware and Software Configuration

[0754] 1. Dedicated terminal: This device is installed in the elderly person's living space (living room, bedroom, etc.) and constantly collects surrounding sounds. This terminal is equipped with a microphone and a communication module for transmitting voice data to a server.

[0755] 2. Generative AI model: A server-based voice analysis system built using machine learning libraries such as TensorFlow and Keras, it analyzes collected voice data to detect abnormal voice patterns (such as calls for help or the sound of falling).

[0756] 3. Smartphone application: This application sends an alert to the user when an abnormal voice pattern is detected. When this application receives an alert, it notifies the user and prompts them to take action.

[0757] Processing flow

[0758] 1. Collection of voice data: A dedicated device collects voice data in the elderly person's living space and transmits it to a server in real time.

[0759] 2. Voice data analysis: A generative AI model on the server analyzes the collected voice data and detects anomalous voice patterns.

[0760] 3. Alerting: If an abnormal voice pattern is detected, the server sends an alert to the smartphone application, which displays a notification to the user, prompting them to take immediate action.

[0761] Specific examples

[0762] For example, if an elderly person falls in the middle of the night, a dedicated device will collect the sound and the generative AI model will recognize it as the "sound of a fall." If an abnormality is detected, the server will send an alert to the user's smartphone saying, "An elderly person has fallen."

[0763] Prompt Sentence Examples

[0764] "Voice data is collected from a dedicated device installed in the elderly's living space and analyzed using a generative AI model. If an abnormal voice pattern (such as a cry for help or the sound of a fall) is detected, an alert is sent to the smartphone."

[0765] In this way, it becomes possible to monitor the safety of elderly people in real time and respond quickly if an abnormality occurs.

[0766] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0767] Step 1:

[0768] The device collects audio in the elderly person's living space. The device uses a microphone to capture ambient audio in real time and converts the data into a digital format. The input is ambient audio, and the output is digital audio data.

[0769] Step 2:

[0770] The terminal transmits the collected digital voice data to the server. The terminal uses a communication module to transmit the collected voice data to the server via the Internet. The input is the digital voice data, and the output is the voice data transmitted to the server.

[0771] Step 3:

[0772] The server inputs the received voice data into a generative AI model. The server uses a generative AI model built using TensorFlow and Keras to analyze the voice data. The input is the voice data sent to the server, and the output is the analysis result.

[0773] Step 4:

[0774] The server detects abnormal voice patterns based on the analysis results of the generative AI model. The server evaluates the analysis results and determines whether they contain abnormal voice patterns such as calls for help or the sound of a fall. The input is the analysis results of the generative AI model, and the output is the anomaly detection results.

[0775] Step 5:

[0776] If the server detects an abnormal voice pattern, it sends an alert to the smartphone application. The server generates an alert message based on the anomaly detection results and sends it to the smartphone. The input is the anomaly detection result, and the output is the alert message.

[0777] Step 6:

[0778] A user receives an alert on a smartphone application. The application notifies the user of the received alert message and prompts them to take action. The input is the alert message, and the output is a notification to the user.

[0779] Example 2

[0780] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0781] Conventional voice analysis systems have difficulty detecting potentially fraudulent conversations in real time, putting users at high risk of becoming victims of fraud. Furthermore, the accuracy of voice data analysis is low, resulting in frequent false positives and false negatives. This creates a need for a highly reliable system to ensure user safety.

[0782] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0783] In this invention, the server includes means for receiving and storing voice data, means for converting the voice data into text, and means for analyzing the text to detect conversations that may be fraudulent, thereby enabling highly accurate analysis of voice data and real-time detection of conversations that may be fraudulent.

[0784] "Audio collection means" means a device or function for recording a user's speech and converting it into digital form.

[0785] The "means for transmitting collected voice to a server" is a device or function for transmitting voice data recorded by the terminal to a server via the Internet.

[0786] The "means for the server to receive and store voice data" refers to a device or function that allows the server to receive voice data sent from the terminal and store it in a specific folder or database.

[0787] The "means for the server to convert voice data into text" refers to a device or function for converting voice data received by the server into text format using natural language processing technology.

[0788] "Means for the server to analyze text and detect potentially fraudulent conversations" refers to a device or function that allows the server to analyze text data and detect specific phrases or patterns to determine whether there is a possibility of fraud.

[0789] "Means for issuing an alert when a potentially fraudulent conversation is detected" refers to a device or function for sending a warning message or notification to a user when the server detects a potentially fraudulent conversation.

[0790] A "dedicated terminal" is an electronic device specialized for collecting and transmitting audio to a server.

[0791] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to convert voice data into text and analyze the text data.

[0792] MODE FOR CARRYING OUT THE INVENTION

[0793] The present invention relates to a system for collecting voice data and detecting potentially fraudulent conversations. A specific embodiment of this system will be described below.

[0794] Audio data collection

[0795] The user uses the terminal to collect voice data. The terminal is equipped with a microphone that records the user's voice. For example, when the user says, "I want to send money," the voice is recorded on the terminal.

[0796] Sending audio data

[0797] The device transmits the collected voice data to a server. This requires an internet connection. Specifically, the device converts the voice data into a digital format and sends it to the server using an HTTP request.

[0798] Receiving and storing audio data

[0799] The server receives the voice data sent from the device. The server temporarily stores the received voice data and prepares it for analysis. For example, the server stores the voice data in a specific folder.

[0800] Converting audio data to text

[0801] The server passes the saved voice data to the chat generation AI, which uses natural language processing technology such as OpenAI's GPT-4. The server inputs the voice data into the AI, which then converts the voice into text.

[0802] Text Analysis

[0803] The server analyzes the converted text to determine whether it is potentially fraudulent. Specifically, the server searches for specific phrases in the text (e.g., "Please send me money") and evaluates whether it is potentially fraudulent.

[0804] Notification of results

[0805] The server sends the analysis results to the device, where the user can check the results. For example, the server sends a message to the device saying "Possible fraud," and the device displays the message to the user.

[0806] Specific examples

[0807] Example 1: Possible fraud

[0808] The user records a voice message saying "I want to send money," and the device sends the message to the server. The server converts the voice message into text and determines that it may be fraudulent. As a result, the device displays a message saying, "This may be fraudulent."

[0809] Example 2: No possibility of fraud

[0810] The user records "Hello, how are you?" and the device sends the audio to the server. The server converts the audio into text and determines that there is no possibility of fraud. As a result, the device displays a message saying "No possibility of fraud."

[0811] Prompt Sentence Examples

[0812] Here are some example prompts to input to a generative AI model:

[0813] "Convert the audio data into text and determine whether it is likely to be fraudulent. For example, if it contains a phrase like 'I want you to send me money,' determine that it is likely to be fraudulent."

[0814] Using this prompt, the generative AI model can properly analyze the voice data and determine the likelihood of fraud.

[0815] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0816] Step 1:

[0817] A user uses a terminal to collect voice data. The user presses the record button on the terminal and says, "I want to send money." The terminal records the voice and converts it into a digital format. The input is the user's voice, and the output is digital voice data.

[0818] Step 2:

[0819] The device sends the collected voice data to the server. The device encodes the voice data in a digital format and sends it to the server using an HTTP request. The input is the digital voice data, and the output is the voice data sent to the server.

[0820] Step 3:

[0821] The server receives and saves the audio data sent from the device. The server receives an HTTP request and saves the audio data in a specific folder. The input is the audio data sent from the device, and the output is the audio data saved on the server.

[0822] Step 4:

[0823] The server passes the voice data it has saved to the chat generation AI. The server inputs the voice data into the AI, which then converts the voice into text. The input is the voice data saved on the server, and the output is text data.

[0824] Step 5:

[0825] The server analyzes the converted text and determines whether it is likely to be fraudulent. The server searches for specific phrases in the text (e.g., "Please send me money") and evaluates whether it is likely to be fraudulent. The input is text data, and the output is the evaluation result regarding the likelihood of fraud.

[0826] Step 6:

[0827] The server sends the analysis results to the terminal. The server sends a message saying "Possible fraud" to the terminal, and the terminal displays the message to the user. The input is the evaluation result regarding the possibility of fraud, and the output is a warning message displayed to the user.

[0828] (Application example 2)

[0829] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0830] In recent years, fraud has become more sophisticated, with voice fraud in particular on the rise. This has increased the risk of many people becoming victims of fraud. Conventional fraud detection systems have the challenge of analyzing voice in real time and quickly determining whether a fraud is likely. Furthermore, there is a need for a system that can monitor the voice calls received by users and surrounding conversations in real time, and immediately issue a warning if there is a possibility of fraud.

[0831] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice, means for analyzing the collected voice and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, means for converting the collected voice into text, means for analyzing the converted text using a generative AI model, and means for inputting a prompt sentence into the generative AI model to determine the possibility of fraud. This makes it possible to monitor voice calls received by the user and surrounding conversations in real time and to immediately issue an alert if there is a possibility of fraud.

[0832] "Means for collecting audio" refers to devices or software for acquiring audio as digital data.

[0833] "Means for analyzing collected audio and detecting potentially fraudulent conversations" refers to devices or software that analyze acquired audio data and determine whether its content constitutes fraudulent activity.

[0834] "Means for issuing an alert when a potentially fraudulent conversation is detected" refers to a device or software that issues a warning to the user when a potentially fraudulent conversation is detected.

[0835] "Means for converting collected voice into text" refers to devices or software for converting voice data into text data.

[0836] "Means for analyzing converted text using a generative AI model" refers to devices or software that analyze text data using a generative AI model and understand its content.

[0837] "Means for inputting prompt sentences into a generative AI model to determine the possibility of fraud" refers to devices or software that input specific prompt sentences into a generative AI model and determine the possibility of fraud based on the response.

[0838] A system for implementing this invention includes means for collecting audio, means for analyzing the collected audio to detect potentially fraudulent conversations, means for issuing an alert when potentially fraudulent conversations are detected, means for converting the collected audio into text, means for analyzing the converted text using a generative AI model, and means for inputting a prompt sentence into the generative AI model to determine the likelihood of fraud.

[0839] Hardware and Software Configuration

[0840] 1. Audio collection method:

[0841] Hardware: Using the microphone in your smartphone or smart glasses.

[0842] Software: For audio collection, we use the speech_recognition library.

[0843] 2. Speech-to-text methods:

[0844] Hardware: Server or cloud service.

[0845] Software: Uses the Google Cloud Speech-to-Text API to convert collected audio into text in real time.

[0846] 3. Analysis methods using generative AI models:

[0847] Hardware: Server or cloud service.

[0848] Software: OpenAI's GPT-4 is used to analyze the converted text.

[0849] 4. Enter prompt text to determine potential fraud:

[0850] Hardware: Server or cloud service.

[0851] Software: Using OpenAI's API, specific prompts are fed into a generative AI model, and the response is used to determine the likelihood of fraud.

[0852] 5. Alert generation method:

[0853] Hardware: Smartphones and smart glasses.

[0854] Software: Applications that notify users of potential fraud.

[0855] Specific examples

[0856] For example, if a user receives a call containing the phrase "send me money," the system works as follows:

[0857] 1. Audio collection: The smartphone microphone collects audio.

[0858] 2. Speech-to-text conversion: The collected audio is converted to text using the Google Cloud Speech-to-Text API.

[0859] 3. Analysis using a generative AI model: The converted text is analyzed using OpenAI's GPT-4.

[0860] 4. Enter the prompt: Enter the following prompt into the generative AI model:

[0861] "Please determine if the following text is a potential scam: Send me money."

[0862] 5. Determining the likelihood of fraud: Determine the likelihood of fraud based on the response of the generative AI model.

[0863] 6. Alert: If a potential fraud attempt is detected, a warning notification will be sent to your smartphone.

[0864] The system allows users to detect potentially fraudulent conversations in real time and respond quickly.

[0865] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0866] Step 1:

[0867] A user collects audio using a smartphone or smart glasses. The input is ambient sound or phone conversation audio, and the output is digital audio data. Specifically, the device's microphone captures the audio and stores it as audio data.

[0868] Step 2:

[0869] The terminal sends the collected voice data to the server. The input is digital voice data, and the output is the voice data sent to the server. Specifically, the terminal's communication module uploads the voice data to the server via the Internet.

[0870] Step 3:

[0871] The server converts the voice data into text using the Google Cloud Speech-to-Text API. The input is digital voice data, and the output is text data. Specifically, the server sends the voice data to the API and receives the returned text data.

[0872] Step 4:

[0873] The server analyzes the converted text data using a generative AI model (OpenAI's GPT-4). The input is text data, and the output is the analysis result. Specifically, the server inputs the text data into the generative AI model and obtains the analysis result.

[0874] Step 5:

[0875] The server inputs a prompt sentence into the generative AI model to determine the likelihood of fraud. The input is the prompt sentence and text data, and the output is the judgment result regarding the likelihood of fraud. Specifically, the server inputs the following prompt sentence into the generative AI model:

[0876] "Please determine if the following text is a potential scam: Send me money."

[0877] Step 6:

[0878] If the server determines that there is a possibility of fraud, it issues an alert to the terminal. The input is the result of the judgment regarding the possibility of fraud, and the output is a warning notification to the user. Specifically, the server sends a notification to the terminal based on the judgment result, and the terminal displays a warning to the user.

[0879] Step 7:

[0880] The user receives the warning and takes the necessary action. The input is the warning notification, and the output is the user's response action. Specifically, the user checks the warning and takes appropriate action against the potentially fraudulent conversation.

[0881] Example 3

[0882] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0883] In recent years, fraud has become more sophisticated, with an increase in fraudulent activities using telephone and voice communications. Such fraudulent activities can cause significant damage, especially to elderly people and those unfamiliar with technology. Conventional methods have made it difficult to detect fraud and prevent damage before it occurs. Therefore, there is a need for a system that can detect potentially fraudulent conversations in real time and issue a prompt alert.

[0884] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[0885] In this invention, the server includes means for collecting voice, means for converting the collected voice into text data using voice recognition technology, means for analyzing the converted text data using a natural language processing model to detect conversations that may be fraudulent, and means for issuing an alert when a conversation that may be fraudulent is detected. This makes it possible to detect conversations that may be fraudulent in real time and to issue an alert quickly.

[0886] The "means for collecting voice" refers to a device or function for acquiring the user's conversation or voice in real time.

[0887] "Speech recognition technology" is a technology for converting acquired voice data into text data, allowing voice to be treated as text information.

[0888] "Text data" is character information converted using speech recognition technology, and is data in a format that can be analyzed and processed.

[0889] A "natural language processing model" is an artificial intelligence model that analyzes text data and determines whether it is likely to be fraudulent.

[0890] A "potentially fraudulent conversation" is a conversation that, after analysis by a natural language processing model, is determined to contain signs of fraudulent activity.

[0891] An "alert issuing means" is a device or function that issues a warning or notification to a user or a third party when a potentially fraudulent conversation is detected.

[0892] A "dedicated device" is a device specialized for collecting audio, including, for example, a smartphone or smart speaker.

[0893] A "generative AI model" is an artificial intelligence model that is used to analyze text data and determine the likelihood of fraud, and is trained for a specific task.

[0894] The present invention relates to a system that detects potentially fraudulent conversations in real time and issues an alert quickly. A specific embodiment of this system will be described below.

[0895] First, the device is equipped with a voice collection means for capturing the user's conversation in real time. The device is equipped with a microphone and records what the user says. Specific hardware used for this is a smartphone or smart speaker.

[0896] The device then converts the captured voice data into text using voice recognition technology. This process uses voice recognition software such as the Google Speech-to-Text API. Once the voice data is converted into text, it can be handled as text information.

[0897] The device then inputs the converted text data into a natural language processing model, specifically a generative AI model such as OpenAI's GPT-4, to analyze the text data and determine whether it is likely to be fraudulent. The model is trained to detect keywords and contexts related to fraud.

[0898] If the device determines that there is a high possibility of fraud based on the analysis results of the NLP model, it sets a fraud detection flag, which triggers the subsequent alert sending process.

[0899] Finally, the device will issue an alert if a fraud detection flag is raised. The alert can take multiple forms, including a voice message from the device or an automated call to registered contacts (e.g., family members or the police). Specifically, an automated call can be made using the Twilio API.

[0900] As a concrete example, consider the case where a user receives a phone call and is told, "Your bank account has been fraudulently used. Please tell us your account number and PIN to verify."

[0901] Process flow:

[0902] 1. The device picks up the user's conversation using a microphone. For example, the microphone on a smartphone records the user's voice.

[0903] 2. The device uses the Google Speech-to-Text API to convert the speech to text, which reads, "Your bank account has been compromised. Please provide your account number and PIN to verify."

[0904] 3. The device inputs the converted text into GPT-4 to analyze the possibility of fraud. GPT-4 detects keywords such as "bank account," "fraudulent use," "account number," and "PIN number," and determines that there is a high possibility of fraud.

[0905] 4. The device determines that there is a high possibility of fraud and sets a fraud detection flag.

[0906] 5. The device automatically calls the user's family using the Twilio API and notifies them that a potentially fraudulent conversation has been detected.

[0907] Example prompts to input to a generative AI model:

[0908] "When a user calls and says, 'Your bank account has been compromised. Please provide your account number and PIN to verify,' analyze whether this is a potential fraud."

[0909] This system makes it possible to prevent fraudulent acts before they occur. The flow of the identification process in the third embodiment will be described with reference to FIG.

[0910] Step 1:

[0911] The device captures the user's conversation in real time. Specifically, it uses a microphone installed on the device to collect voice data. The input is the user's voice, and the output is digital voice data. This voice data is used in subsequent processing steps.

[0912] Step 2:

[0913] The device converts the collected voice data into text data using voice recognition technology. Specifically, it uses the Google Speech-to-Text API to analyze the voice data and generate corresponding text data. The input is voice data and the output is text data. This text data is used in subsequent natural language processing steps.

[0914] Step 3:

[0915] The device inputs the converted text data into a natural language processing model. Specifically, it uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the text data and determine whether it is likely to be fraud. The input is the text data, and the output is an analysis result indicating the likelihood of fraud. This analysis result is used to raise a fraud detection flag.

[0916] Step 4:

[0917] If the device determines that there is a high possibility of fraud based on the analysis results of the NLP model, it sets a fraud detection flag. The input is the analysis results of the NLP model, and the output is a fraud detection flag. This flag triggers the subsequent alert sending process.

[0918] Step 5:

[0919] If a fraud detection flag is raised, the device will issue an alert. Specifically, this can be a voice message from the device or an automated call to registered contacts (e.g., family members or the police). The input is the fraud detection flag and the output is an alert notification. Specific actions include making an automated call using the Twilio API.

[0920] (Application example 3)

[0921] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0922] In recent years, fraud has been on the rise, particularly over the phone. This has resulted in many people suffering financial losses. Conventional methods have made it difficult to detect potentially fraudulent conversations in real time and respond quickly. Therefore, there is a need for a system that can detect potentially fraudulent conversations in real time, quickly issue alerts, and, if necessary, automatically communicate with registered contacts.

[0923] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[0924] In this invention, the server includes means for collecting audio, means for analyzing the collected audio and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, and means for automatically communicating with registered contacts when a conversation that may be fraudulent is detected. This makes it possible to detect conversations that may be fraudulent in real time and respond quickly.

[0925] "Means for collecting audio" refers to devices and software for acquiring audio data.

[0926] "Means for analyzing collected audio and detecting potentially fraudulent conversations" refers to devices or software that analyze acquired audio data and identify content that may be fraudulent.

[0927] An "alert mechanism" is a device or software that issues a warning when it detects a potentially fraudulent conversation.

[0928] "Means for automatically communicating with registered contacts" refers to devices or software that automatically communicate with pre-registered contacts when a conversation that may be fraudulent is detected.

[0929] A "purpose-built terminal" is an electronic device designed for a specific purpose.

[0930] A "generative AI model" is a model generated using artificial intelligence techniques and trained to perform a specific task.

[0931] A system for implementing this invention includes means for collecting audio, means for analyzing the collected audio and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, and means for automatically communicating with registered contacts when a conversation that may be fraudulent is detected. Specific embodiments are described below.

[0932] Hardware and Software Configuration

[0933] The system is implemented using mobile devices such as smartphones. It uses the smartphone's microphone to collect audio. It uses the Python programming language and the SpeechRecognition library to analyze the audio data. It uses a generative AI model to detect potentially fraudulent conversations. It uses the Twilio API to trigger voice messages and automated phone calls to issue alerts.

[0934] Data processing and calculation

[0935] The device first collects voice data through a microphone. The collected voice data is converted to text data using the SpeechRecognition library. The text data is then analyzed using a generative AI model to detect potentially fraudulent keywords and phrases. If a potentially fraudulent conversation is detected, the device uses the Twilio API to alert the user with a voice message and automatically make a call to a registered contact.

[0936] Specific examples

[0937] For example, if a user says, "Please tell me my bank account information," the system will detect the keyword "bank account," issue an alert, and automatically call a registered contact.

[0938] Prompt Sentence Examples

[0939] "Create an application that detects potentially fraudulent conversations and alerts users. Include functionality to capture voice data and trigger alerts if fraudulent keywords are detected."

[0940] In this way, the present invention is able to detect potentially fraudulent conversations in real time and respond quickly.

[0941] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[0942] Step 1:

[0943] The device collects voice data using the smartphone's microphone. When the user starts a call, the device continues to capture voice data in real time. The input is the user's voice, and the output is the collected voice data.

[0944] Step 2:

[0945] The device converts the collected voice data into text data using the SpeechRecognition library. To convert voice data into text data, it uses Google's speech recognition API. The input is voice data, and the output is the converted text data.

[0946] Step 3:

[0947] The device analyzes text data using a generative AI model to detect potentially fraudulent keywords and phrases. The generative AI model scans the text data using a pre-trained list of fraudulent keywords. The input is the text data, and the output is a judgment of whether it is potentially fraudulent.

[0948] Step 4:

[0949] If a potentially fraudulent conversation is detected, the device uses the Twilio API to alert the user with a voice message. The input is the result of the potentially fraudulent conversation, and the output is an audio warning to the user.

[0950] Step 5:

[0951] Furthermore, if a potentially fraudulent conversation is detected, the device will automatically call registered contacts using the Twilio API. The input is the result of the potentially fraudulent conversation, and the output is an automated call to the registered contact.

[0952] In this way, the device can detect potentially fraudulent conversations in real time and respond quickly.

[0953] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0954] "Example 1"

[0955] In one embodiment of the present invention, the system includes an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's voice and issues an alert based on the user's emotional state. For example, if the user expresses negative emotions such as anger or fear, the system determines that fraud is a possibility and issues an alert.

[0956] "Example 2"

[0957] In addition, the emotion engine strengthens the detection of potentially fraudulent conversations when the user's emotional state deteriorates above a certain threshold. For example, if the user shows strong anger or fear, the emotion engine will detect that emotional state and strengthen the detection of potentially fraudulent conversations. This allows for more accurate detection of situations in which the user is likely to be scammed.

[0958] "Example 3"

[0959] Furthermore, the emotion engine monitors the user's emotional state in real time and controls the issuance of alerts according to changes in the emotional state. For example, if the user's emotion changes from anger or fear to relief or joy, the emotion engine detects this change in emotional state and stops issuing alerts. This allows the system to provide appropriate alerts according to the user's emotional state.

[0960] The processing flow of each embodiment will be described below.

[0961] "Example 1"

[0962] Step 1: Collect the user's voice using a dedicated device.

[0963] Step 2: The collected voice data is sent to the emotion engine to recognize the user's emotions.

[0964] Step 3: Detect potentially fraudulent conversations based on the emotional state recognized by the emotion engine.

[0965] Step 4: If a potentially fraudulent conversation is detected, issue an alert.

[0966] "Example 2"

[0967] Step 1: Collect the user's voice using a dedicated device.

[0968] Step 2: The collected voice data is sent to the emotion engine to recognize the user's emotions.

[0969] Step 3: Enhance detection of potentially fraudulent conversations when the emotional state recognized by the emotion engine deteriorates above a certain threshold.

[0970] Step 4: If a potentially fraudulent conversation is detected, issue an alert.

[0971] "Example 3"

[0972] Step 1: Collect the user's voice using a dedicated device.

[0973] Step 2: The collected voice data is sent to the emotion engine to recognize the user's emotions.

[0974] Step 3: Monitor the emotional state recognized by the emotion engine in real time and control the issuance of alerts according to changes in the emotional state.

[0975] Step 4: If the user's emotional state changes to a positive state, such as relief or joy, stop issuing alerts.

[0976] Example 1

[0977] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0978] In order to reduce the risk of elderly people falling victim to fraud, it is important to detect signs of fraud early in everyday conversations and take appropriate action. However, conventional systems lack sufficient analysis of voice data and emotion recognition, making it difficult to accurately determine the possibility of fraud. In addition, there is a lack of means to evaluate the user's emotional state, which can lead to delayed response when the user displays negative emotions. This has led to the issue of increasing the risk of elderly people falling victim to fraud.

[0979] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0980] In this invention, the server includes means for collecting voice, means for analyzing the collected voice to detect conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, means for converting voice data into text data, means for recognizing emotions from the text data, and means for evaluating the user's emotional state based on the emotion recognition results. This makes it possible to accurately recognize emotions from voice data and detect the possibility of fraud early. Furthermore, by evaluating the user's emotional state, it is possible to respond quickly when negative emotions are displayed, thereby reducing the risk of elderly people being scammed.

[0981] "Audio collection means" refers to a device or equipment for capturing ambient sounds, particularly a dedicated terminal that includes a highly sensitive microphone.

[0982] "Means for analyzing collected audio to detect potentially fraudulent conversations" refers to algorithms or software that analyze voice data and identify signs of fraud from its content.

[0983] "Means for issuing alerts when potentially fraudulent conversations are detected" refers to systems or features that send notifications to designated contacts when a potential fraudulent conversation is detected.

[0984] "Means for converting voice data into text data" refers to software or algorithms for converting voice data into text information using voice recognition technology.

[0985] "Means for recognizing emotions from text data" refers to emotion analysis algorithms or software that analyze text data and identify the user's emotions from its content.

[0986] "Means for evaluating the user's emotional state based on the emotion recognition results" refers to a system or function that comprehensively evaluates the user's emotional state based on the results of emotion analysis and takes appropriate measures as necessary.

[0987] This invention is a system for reducing the risk of elderly people falling victim to fraud, and by collecting and analyzing voice and recognizing emotions, it can detect the possibility of fraud early and issue an alert. A specific embodiment of this system will be described below.

[0988] Audio Collection

[0989] The device is installed in a place frequently used by the elderly, such as the living room or bedroom. The device is equipped with a highly sensitive microphone that constantly collects surrounding sounds. For example, the device can capture the elderly's voice saying, "Hello, how was your day?"

[0990] Sending audio data

[0991] The device transmits the collected audio data to the server in real time. The audio data is encrypted and transmitted using a secure communication protocol (e.g., HTTPS). For example, the device encrypts the audio data it captures and transmits it to the server using the HTTPS protocol.

[0992] Analysis of audio data

[0993] The server uses speech recognition software (e.g., Google Cloud Speech-to-Text API) to analyze the received voice data. The server converts the voice data into text. For example, the voice data "Hello, how was your day?" is converted into text data "Hello, how was your day?"

[0994] emotion recognition

[0995] The server inputs the text data into an emotion analysis algorithm (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. The emotion analysis algorithm identifies the type of emotion (e.g., anger, fear, joy, etc.) from the text data. For example, the phrase "Hello, how was your day?" is analyzed from the text data, and the emotion is recognized as "neutral."

[0996] Alert sending

[0997] The server evaluates the user's emotional state based on the results of emotion analysis. If the user shows negative emotions such as anger or fear, the server determines that there is a possibility of fraud and issues an alert. The alert is sent to designated contacts (e.g., family members or caregivers) via email or SMS. For example, if the user utters the text data "Help me, I'm scared!", the server recognizes the emotion as "fear" and determines that there is a possibility of fraud. The server then sends an email alert to the family saying, "Mom is scared. Please check on her."

[0998] Prompt Sentence Examples

[0999] An example of a prompt to be input to the generative AI model is as follows:

[1000] "Describe a system that collects audio data from elderly people having conversations in their living rooms and recognizes emotions from the audio. Include the ability to issue an alert if the user is expressing anger or fear."

[1001] Using this prompt, the generative AI model can generate a detailed description of the system.

[1002] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1003] Step 1:

[1004] The device is installed in a place frequently used by the elderly, such as the living room or bedroom. The device uses a highly sensitive microphone to constantly collect surrounding sounds. The input is the surrounding sounds, and the output is the collected voice data. For example, the device captures the elderly person's voice saying, "Hello, how was your day?"

[1005] Step 2:

[1006] The device transmits the collected audio data to the server in real time. The audio data is encrypted and transmitted using a secure communication protocol (e.g., HTTPS). The input is the collected audio data, and the output is the encrypted audio data. For example, the device encrypts the audio data it captures and transmits it to the server using the HTTPS protocol.

[1007] Step 3:

[1008] The server uses speech recognition software (e.g., Google Cloud Speech-to-Text API) to analyze the received voice data. The server converts the voice data into text. The input is encrypted voice data, and the output is text data. For example, the voice data "Hello, how was your day?" is converted into text data "Hello, how was your day?"

[1009] Step 4:

[1010] The server inputs the text data into an emotion analysis algorithm (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. The emotion analysis algorithm identifies the type of emotion (e.g., anger, fear, joy, etc.) from the text data. The input is the text data, and the output is the emotion recognition result. For example, the phrase "Hello, how was your day?" is analyzed from the text data, and the emotion is recognized as "neutral."

[1011] Step 5:

[1012] The server evaluates the user's emotional state based on the results of emotion analysis. If the user shows negative emotions such as anger or fear, the server determines that there is a possibility of fraud and issues an alert. The input is the emotion recognition result, and the output is an alert notification. For example, if the user utters text data such as "Help, I'm scared!", the server recognizes the emotion as "fear" and determines that there is a possibility of fraud. The server then sends an email alert to the family saying, "Mom is scared. Please check on her."

[1013] (Application example 1)

[1014] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1015] In order to reduce the risk of elderly people becoming victims of fraud, it is necessary to detect potentially fraudulent conversations in real time and respond quickly. However, conventional systems have issues with not being able to completely prevent fraud because they are unable to collect and analyze voice data properly and are unable to send alerts that take into account changes in emotions.

[1016] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for collecting voice, means for analyzing the collected voice and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, means for recognizing emotions, and means for issuing an alert when the recognized emotions are negative. This reduces the risk of elderly people becoming victims of fraud and enables prompt response.

[1017] "Voice collection means" refers to devices or software for collecting the user's voice in real time.

[1018] "Means for analyzing collected audio to detect potentially fraudulent conversations" refers to algorithms or programs that analyze collected audio data and identify potentially fraudulent conversations from their content.

[1019] "Means for issuing an alert when a potentially fraudulent conversation is detected" refers to a device or software that issues a warning to a user or a third party when a potentially fraudulent conversation is detected.

[1020] "Means for recognizing emotions" refers to algorithms or programs that analyze a user's emotions from voice data and recognize a specific emotional state.

[1021] "Means for issuing an alert when the recognized emotion is negative" refers to a device or software for issuing a warning to a user or a third party when the recognized emotion is a negative state such as anger or fear.

[1022] "Specialized Device" means a hardware device designed and manufactured for a specific purpose.

[1023] A "generative AI model" is a model generated using artificial intelligence techniques and refers to an algorithm trained to perform a specific task.

[1024] This invention is a system for reducing the risk of elderly people becoming victims of fraud, and it has the ability to collect voice, recognize emotions, and detect potentially fraudulent conversations. The system is realized using a dedicated device, an emotion recognition engine, and a generative AI model.

[1025] System configuration

[1026] The system consists of the following main components:

[1027] 1. Dedicated terminal: A hardware device that is installed in a location frequently used by elderly people to collect their voice. This terminal has a built-in microphone and collects voice in real time.

[1028] 2. Emotion Recognition Engine: Software that analyzes user emotions from collected voice data and recognizes specific emotional states. This engine takes voice data as input and outputs an emotional state.

[1029] 3. Generative AI model: An algorithm that analyzes collected voice data and detects potentially fraudulent conversations. This model takes voice data as input and determines whether it is likely to be fraudulent.

[1030] 4. Alerting system: Software that warns users and third parties when potentially fraudulent conversations or negative sentiment are detected. Alerts are sent via email and smartphone notifications.

[1031] Program processing explanation

[1032] The server collects voice data from a dedicated device and analyzes it using an emotion recognition engine and generative AI model. Specifically, the following process is carried out:

[1033] 1. Voice collection: A dedicated device collects the voice of the elderly person in real time. The collected voice data is sent to a server.

[1034] 2. Emotion Recognition: The server uses an emotion recognition engine to analyze the user's emotions from the voice data. For example, if the user expresses anger or fear, the emotional state is recognized.

[1035] 3. Fraud Detection: The server analyzes the audio data using a generative AI model to detect potentially fraudulent conversations. If fraud is detected, the information is sent to an alert system.

[1036] 4. Alerting: When potentially fraudulent conversations or negative sentiment are detected, the alerting system will warn users or third parties via email or smartphone notifications.

[1037] Specific examples

[1038] For example, if an elderly person is talking to a scammer on the phone, and emotions such as "anger" or "fear" are detected in the conversation, an alert email will be sent to the family stating, "An elderly person is showing signs of anger. There is a possibility of fraud."

[1039] Prompt Sentence Examples

[1040] "Develop a system that collects voices from elderly people and recognizes their emotions. If emotions such as anger or fear are detected in the voice, the application will send an alert email to their family members. The libraries used are speech_recognition, emotion_recognition, and smtplib."

[1041] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1042] Step 1:

[1043] A dedicated terminal collects the elderly person's voice in real time. The terminal's microphone captures the voice data and sends it to a server. The input is the elderly person's voice, and the output is the voice data sent to the server.

[1044] Step 2:

[1045] The server inputs the received voice data into an emotion recognition engine, which analyzes the voice data and identifies the user's emotional state. The input is the voice data, and the output is the recognized emotional state (e.g., anger, fear).

[1046] Step 3:

[1047] The server evaluates the emotional state obtained from the emotion recognition engine, and if a negative emotion (e.g., anger, fear) is detected, it sends the information to the alerting system. The input is the recognized emotional state, and the output is the emotional information sent to the alerting system.

[1048] Step 4:

[1049] The server inputs the voice data into a generative AI model, which analyzes the voice data and detects potentially fraudulent conversations. The input is the voice data, and the output is a judgment result on whether or not there is a possibility of fraud.

[1050] Step 5:

[1051] The server evaluates the judgment results obtained from the generative AI model, and if it determines that there is a possibility of fraud, it sends that information to the alert system. The input is the judgment result of whether there is a possibility of fraud, and the output is the fraud information sent to the alert system.

[1052] Step 6:

[1053] The alert system issues warnings to users and third parties based on emotional information and fraud information received from the server. The warnings are sent via email or smartphone notifications. The input is emotional information and fraud information, and the output is a warning message sent to users and third parties.

[1054] Example 2

[1055] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1056] While conventional voice analysis systems can detect potentially fraudulent conversations, they have the problem of making it difficult to accurately assess the risk of fraud because they do not take the user's emotional state into account. Furthermore, they are unable to adequately detect situations in which the likelihood of fraud increases when the user is expressing strong emotions, which makes it difficult to adequately ensure the user's safety.

[1057] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1058] In this invention, the server includes means for collecting audio, means for analyzing the collected audio to detect potentially fraudulent conversations, means for issuing an alert when potentially fraudulent conversations are detected, means for detecting the user's emotional state, and means for enhancing the detection of potentially fraudulent conversations when the user's emotional state deteriorates to a certain threshold or greater, thereby enabling accurate determination of fraud risk taking the user's emotional state into account.

[1059] A "means for collecting voice" is a device or software for acquiring voice data provided by a user in digital form and transmitting it to a server.

[1060] "Means for analyzing collected audio to detect potentially fraudulent conversations" refers to technology that converts audio data into text and analyzes that text to identify potentially fraudulent phrases and patterns.

[1061] The "means for issuing an alert when a potentially fraudulent conversation is detected" refers to a device or software for issuing a warning to a user when a potentially fraudulent conversation is detected.

[1062] "Means for detecting the user's emotional state" refers to technology for analyzing the user's voice and facial expression data to evaluate their emotional state.

[1063] "Means for strengthening the detection of potentially fraudulent conversations when a user's emotional state deteriorates above a certain threshold" refers to technology that more strictly detects potentially fraudulent conversations when a user is expressing strong emotions such as anger or fear.

[1064] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to analyze text data and identify specific patterns and phrases.

[1065] A "prompt sentence" is text data that is input into a generative AI model and is a sentence that specifically indicates the content of the object to be analyzed.

[1066] The present invention relates to a system for collecting voice data and detecting potentially fraudulent conversations. A specific embodiment of this system will be described below.

[1067] Audio data collection

[1068] The user provides voice data. The device collects this voice using a built-in microphone and converts it into a digital format. The device then transmits the collected voice data to a server.

[1069] Converting audio data to text

[1070] The server receives the voice data sent from the device. The server calls the Google Cloud Speech-to-Text API and converts the voice data into text. For example, the text generated is "Hello, how are you doing lately?"

[1071] Detecting potential fraud

[1072] The server inputs the generated text into a generative AI model (e.g., OpenAI's GPT-4). The generative AI model uses natural language processing techniques to analyze whether the text is likely to be fraudulent. For example, if the text contains the phrase "please send me money," it will determine that it is likely to be fraudulent.

[1073] Emotional state detection

[1074] The device collects the user's voice and facial expression data and sends it to a server, which then uses Affectiva's emotion recognition API to analyze the user's emotional state. For example, if the user is expressing strong anger or fear, the server can detect that emotional state.

[1075] Enhanced fraud detection

[1076] The server will strengthen its detection of potentially fraudulent conversations if the user's emotional state deteriorates above a certain threshold. For example, if the user expresses strong fear when asked to "send money," it will determine that the conversation is likely to be fraudulent.

[1077] Notification of results

[1078] If the server determines that there is a possibility of fraud, it notifies the terminal of the result. The terminal displays a warning to the user, saying, "There is a possibility of fraud. Please be careful."

[1079] Specific examples

[1080] The user says, "I want you to send me money." The device collects this audio and sends it to the server. The server converts the audio into text and inputs it into a generative AI model. The generative AI model determines that there is a possibility of fraud. At the same time, the device collects the user's facial expression data and sends it to the server. The server uses Affectiva's emotion recognition API to detect that the user is showing strong fear. The server determines that there is a high possibility of fraud and notifies the device to display a warning. The device displays a message to the user saying, "This may be a scam. Please be careful."

[1081] Prompt Sentence Examples

[1082] "Please help us determine if the following text is a potential scam: 'I want you to send me money'"

[1083] This system is expected to protect users from the risk of fraud.

[1084] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1085] Step 1:

[1086] The user provides voice data. The device uses the built-in microphone to collect this voice and convert it into digital form. The input is the user's voice data, and the output is digital voice data. In concrete terms, the device captures voice through the microphone and performs digital signal processing.

[1087] Step 2:

[1088] The terminal transmits the collected digital voice data to the server. The input is the digital voice data, and the output is the voice data transmitted to the server. Specifically, the terminal uploads the voice data to the server via a network.

[1089] Step 3:

[1090] The server receives the voice data sent from the terminal. The input is the voice data sent from the terminal, and the output is the voice data stored in the server. In concrete terms, the server receives the voice data via the network and stores it in a database.

[1091] Step 4:

[1092] The server calls the Google Cloud Speech-to-Text API to convert the voice data to text. The input is the voice data stored on the server, and the output is text data. Specifically, the server calls the API and sends a request to convert the voice data to text.

[1093] Step 5:

[1094] The server inputs the generated text data into a generative AI model (for example, OpenAI's GPT-4) and analyzes whether there is a possibility of fraud. The input is text data, and the output is the analysis result regarding the possibility of fraud. Specifically, the server inputs the text data into the generative AI model and obtains the analysis result.

[1095] Step 6:

[1096] The device collects the user's voice and facial expression data and sends it to the server. The input is the user's voice and facial expression data, and the output is the emotion data sent to the server. Specifically, the device uses a camera and microphone to capture the user's facial expressions and voice, converts them into digital format, and sends them to the server.

[1097] Step 7:

[1098] The server uses Affectiva's emotion recognition API to analyze the user's emotional state. The input is the emotional data sent to the server, and the output is the analysis result of the user's emotional state. Specifically, the server calls the emotion recognition API and sends a request to analyze the emotional data.

[1099] Step 8:

[1100] The server enhances detection of potentially fraudulent conversations when the user's emotional state deteriorates above a certain threshold. The input is the analysis result of the user's emotional state, and the output is the enhanced fraud detection result. Specifically, the server evaluates the analysis result of the emotional state and adjusts the threshold of the fraud detection algorithm.

[1101] Step 9:

[1102] If the server determines that there is a possibility of fraud, it notifies the terminal of the result. The input is the enhanced fraud detection result, and the output is a warning message sent to the terminal. Specifically, the server generates a warning message and sends it to the terminal via the network.

[1103] Step 10:

[1104] The terminal displays a warning to the user, informing them that "This may be a scam. Please be careful." The input is the warning message sent from the server, and the output is the warning message displayed to the user. Specifically, the terminal displays the warning message on the screen and notifies the user.

[1105] (Application example 2)

[1106] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1107] In recent years, telephone and internet-based fraud has been on the rise, with many people falling victim to it. Elderly people and those unfamiliar with technology are particularly vulnerable targets, and fraud methods are becoming more sophisticated, making it difficult to adequately counter these attacks using conventional methods. Furthermore, it is necessary to not only detect potentially fraudulent conversations, but also to more accurately assess the likelihood of fraud by taking into account the user's emotional state. This will reduce the risk of users falling victim to fraud and provide safer communications.

[1108] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1109] In this invention, the server includes means for collecting audio, means for analyzing the collected audio to detect potentially fraudulent conversations, means for issuing an alert when potentially fraudulent conversations are detected, means for detecting a user's emotional state, and means for enhancing detection of potentially fraudulent conversations when the user's emotional state deteriorates to a certain threshold or greater. This makes it possible to detect potentially fraudulent conversations in real time and issue a warning taking the user's emotional state into consideration.

[1110] The "means for collecting voice" is a device or software for capturing the user's voice in real time.

[1111] "Means for analyzing collected audio to detect potentially fraudulent conversations" refers to technology that analyzes acquired audio data and detects phrases or patterns that may be fraudulent.

[1112] The "means for issuing an alert when a potentially fraudulent conversation is detected" is a device or software for issuing a warning to a user when a potentially fraudulent conversation is detected.

[1113] The "means for detecting the user's emotional state" is a technology for analyzing and detecting the user's emotional state from their voice or other data.

[1114] "Means for strengthening detection of potentially fraudulent conversations when a user's emotional state deteriorates beyond a certain threshold" refers to technology for more rigorously detecting potentially fraudulent conversations when a user's emotional state deteriorates.

[1115] A system for implementing the present invention includes a set of means for collecting audio, detecting potentially fraudulent conversations, and issuing a warning to the user. A specific embodiment of this system is described below.

[1116] First, the smartphone's microphone is used to collect audio. The smartphone monitors the user's calls in real time and collects audio data. The collected audio data is converted into text using speech recognition software (e.g., the speech_recognition library).

[1117] Next, a chat generation AI model (for example, a model using the transformers library) is used to analyze the collected audio and detect potentially fraudulent conversations. This model analyzes the converted text and detects phrases and patterns that may be fraudulent. For example, if a phrase such as "please send money" is included, it is determined to be fraudulent.

[1118] Furthermore, emotion recognition software (e.g., emotion_recognition library) is used as a means of detecting the user's emotional state. This software analyzes and detects the user's emotional state from voice and text data. If the user shows strong anger or fear, it is determined that the user's emotional state is deteriorating.

[1119] The system will integrate emotion recognition software with chat generation AI models as a means of strengthening detection of potentially fraudulent conversations when a user's emotional state deteriorates above a certain threshold. When the user's emotional state deteriorates, the system will more strictly detect potentially fraudulent conversations.

[1120] The system uses the smartphone's notification function to issue an alert when a conversation that may be fraudulent is detected. When a conversation that may be fraudulent is detected, a warning is issued to the user to warn them.

[1121] As a concrete example, the following prompt sentence is input into the generative AI model:

[1122] User: Hello, how are you doing?

[1123] Scammer: Hello, I'm actually in a bit of a pinch and I'd like you to send me some money.

[1124] When the user enters this prompt, the chat generation AI model will determine that it is a potential scam and issue a warning to the user. If the user expresses strong anger or fear, emotion recognition software will detect that emotional state and reinforce the warning.

[1125] In this way, potentially fraudulent conversations can be detected in real time and alerted taking into account the user's emotional state.

[1126] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1127] Step 1:

[1128] The device collects the user's voice. Specifically, it uses the smartphone's microphone to capture voice data during a call in real time. The input is the user's voice, and the output is voice data.

[1129] Step 2:

[1130] The device converts the collected voice data into text. Specifically, it uses the speech_recognition library to perform voice recognition. The input is voice data, and the output is text data.

[1131] Step 3:

[1132] The server analyzes the converted text data and detects potentially fraudulent conversations. Specifically, it uses the transformers library to input text data into a chat generation AI model to detect potentially fraudulent phrases and patterns. The input is text data, and the output is a judgment result on whether or not there is a possibility of fraud.

[1133] Step 4:

[1134] The server detects the user's emotional state. Specifically, it analyzes the emotional state from text data using the emotion_recognition library. The input is text data, and the output is the analysis result of the emotional state.

[1135] Step 5:

[1136] The server strengthens its detection of potentially fraudulent conversations when the user's emotional state deteriorates above a certain threshold. Specifically, when the emotional state is strong, such as "anger" or "fear," it more strictly detects potentially fraudulent phrases and patterns. The input is the analysis result of the emotional state, and the output is the judgment result of the enhanced fraud detection.

[1137] Step 6:

[1138] The device issues an alert when a potentially fraudulent conversation is detected. Specifically, it displays a warning to the user using the smartphone's notification function. The input is the fraud detection result, and the output is a warning notification to the user.

[1139] Example 3

[1140] Next, a description will be given of Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1141] Conventional fraud detection systems focus on detecting potentially fraudulent conversations, but because they do not take into account the user's emotional state, they may issue alerts at the wrong time. Furthermore, they lack the ability to appropriately control alerts when the user's emotional state changes, which can cause unnecessary stress to the user. This creates a problem of impairing the user experience.

[1142] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1143] In this invention, the server includes means for collecting voice, means for analyzing the collected voice and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, means for monitoring the user's emotional state in real time, and means for controlling the issuance of the alert in response to changes in the user's emotional state. This makes it possible not only to detect conversations that may be fraudulent, but also to provide appropriate alerts that correspond to the user's emotional state.

[1144] "Voice collection means" refers to a device or function for acquiring the user's voice and storing it as digital data.

[1145] "Means for analyzing collected audio and detecting potentially fraudulent conversations" refers to devices or functions that convert acquired audio data into text data and analyze that text data to detect keywords or phrases that may be fraudulent.

[1146] "Means for issuing an alert when a potentially fraudulent conversation is detected" means a device or function for issuing a warning to a user when a potentially fraudulent conversation is detected, and includes forms such as a voice message or automated phone call.

[1147] The "means for monitoring the user's emotional state in real time" refers to a device or function for analyzing the user's voice data and grasping the user's emotional state in real time.

[1148] The "means for controlling the generation of an alert in response to a change in the user's emotional state" refers to a device or function for appropriately controlling the generation of an alert in response to a change in the user's emotional state.

[1149] MODE FOR CARRYING OUT THE INVENTION

[1150] The present invention is a system for detecting potentially fraudulent conversations and providing an appropriate alert to the user. A specific embodiment of this system will be described below.

[1151] Hardware and software used

[1152] The system uses a microphone to receive voice input, a speaker or phone-enabled device to generate alerts, and software including a speech recognition engine, emotion engine, and alert management system. The software uses Google Cloud Speech-to-Text API for speech recognition and IBM Watson Sentiment Analysis API for emotion analysis.

[1153] Data processing and calculation

[1154] 1. Acquiring voice input:

[1155] When the user speaks, the terminal acquires voice data through a microphone.

[1156] What it does: The device's microphone captures the user's voice and stores it as digital audio data.

[1157] 2. Speech Recognition:

[1158] The device sends the acquired voice data to the Google Cloud Speech-to-Text API and converts it into text data.

[1159] Specific operation: The device uploads the voice data to the cloud and receives the text data returned by the API.

[1160] 3. Fraud detection:

[1161] The device then analyzes the converted text data to detect potentially fraudulent keywords and phrases.

[1162] What it does: The device feeds the text data into an internal fraud detection algorithm to detect risky phrases like "Can you tell me your bank account number?"

[1163] 4. Emotional state monitoring:

[1164] The device sends the user's voice data to IBM Watson's emotion analysis API to monitor the user's emotional state in real time.

[1165] How it works: The device uploads voice data to the cloud and receives emotional state data returned by the API, determining whether the user is feeling angry or scared, for example.

[1166] 5. Alert occurs:

[1167] The device will issue an alert if it detects a potentially fraudulent conversation, which can take the form of a voice message or automated phone call.

[1168] Specific behavior: The device will issue a voice warning saying "This may be a scam. Please be careful." If necessary, it will also automatically call registered contacts (for example, family members or the police).

[1169] The emotion engine will stop issuing alerts if the user's emotional state changes to relief or joy.

[1170] Specific behavior: The emotion engine monitors the user's emotional state and stops issuing alerts if emotions such as relief or joy are detected.

[1171] Specific examples

[1172] 1. Detecting potentially fraudulent conversations:

[1173] Scenario: A user calls and says, "What's your bank account number?"

[1174] process:

[1175] The device captures the audio and converts it into text using the Google Cloud Speech-to-Text API.

[1176] The device analyzes the text data and detects the phrase "Please tell me your bank account number."

[1177] The device will issue a voice warning saying, "This may be a scam. Please be careful."

[1178] 2. Controlling alerts based on changes in emotional state:

[1179] Scenario: The user starts out feeling angry or scared, but as the conversation progresses, they begin to feel relieved or happy.

[1180] process:

[1181] The device sends voice data to IBM Watson's sentiment analysis API to monitor the user's emotional state.

[1182] The emotion engine will stop issuing alerts if the user's emotional state changes to relief or joy.

[1183] Prompt Sentence Examples

[1184] Prompt: "Describe a system that detects potentially fraudulent conversations and alerts the user. Include the names of specific hardware and software, and explain how the data is processed and calculated. Include examples."

[1185] In this way, an embodiment of a system that detects conversations that may be fraudulent and provides an appropriate alert to the user will be described. The flow of the identification process in Example 3 will be described with reference to FIG.

[1186] Program processing flow

[1187] Step 1: Getting voice input

[1188] When the user speaks, the terminal acquires voice data through a microphone.

[1189] Input: User's voice

[1190] What it does: The device's microphone captures the user's voice and stores it as digital audio data.

[1191] Output: Digital audio data

[1192] Step 2: Voice Recognition

[1193] The terminal transmits the acquired voice data to a voice recognition engine and converts it into text data.

[1194] Input: Digital audio data

[1195] Specific operation: The device uploads voice data to the cloud and receives text data returned by a speech recognition engine (e.g., Google Cloud Speech-to-Text API).

[1196] Output: Text data

[1197] Step 3: Fraud detection

[1198] The device then analyzes the converted text data to detect potentially fraudulent keywords and phrases.

[1199] Input: Text data

[1200] What it does: The device feeds the text data into an internal fraud detection algorithm to detect risky phrases like "Can you tell me your bank account number?"

[1201] Output: Potentially fraudulent phrase detection results

[1202] Step 4: Monitor your emotional state

[1203] The terminal transmits the user's voice data to an emotion analysis engine to monitor the user's emotional state in real time.

[1204] Input: Digital audio data

[1205] Specific operation: The device uploads voice data to the cloud and receives emotional state data returned by a sentiment analysis engine (e.g., IBM Watson sentiment analysis API), determining, for example, whether the user is feeling angry or scared.

[1206] Output: Emotional state data

[1207] Step 5: Raise an alert

[1208] The terminal will issue an alert if a potentially fraudulent conversation is detected.

[1209] Input: Possible fraud phrase detection results, emotional state data

[1210] Specific behavior: The device will issue a voice warning saying "This may be a scam. Please be careful." If necessary, it will also automatically call registered contacts (for example, family members or the police).

[1211] Output: Alert generation (voice message, automated phone call)

[1212] Step 6: Controlling alerts due to changes in emotional state

[1213] The emotion engine will stop issuing alerts if the user's emotional state changes to relief or joy.

[1214] Input: Emotional state data

[1215] Specific behavior: The emotion engine monitors the user's emotional state and stops issuing alerts if emotions such as relief or joy are detected.

[1216] Output: Alert stopped

[1217] In this way, the process of the system that detects potentially fraudulent conversations and provides appropriate alerts to the user will be described in detail.

[1218] (Application example 3)

[1219] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1220] In recent years, fraud has become more sophisticated, especially with the rise of voice-based fraud. This has increased the risk of many people becoming victims of fraud. Furthermore, while people are becoming more vigilant against fraud, users' emotional states are often unstable. While conventional systems can detect potentially fraudulent conversations, they have the problem of being unable to respond appropriately based on the user's emotional state.

[1221] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1222] In this invention, the server includes means for collecting voice, means for analyzing the collected voice to detect conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, and means for monitoring the user's emotional state in real time and controlling the issuance of the alert in response to changes in the emotional state, thereby making it possible to prevent fraudulent acts and provide appropriate alerts in accordance with the user's emotional state.

[1223] The "means for collecting voice" refers to a device or software for acquiring the user's voice in real time.

[1224] "Means for analyzing collected audio to detect potentially fraudulent conversations" refers to algorithms or generative AI models that analyze acquired audio data and identify potentially fraudulent phrases or patterns.

[1225] "Means for issuing alerts when potentially fraudulent conversations are detected" refers to a voice message or automated phone call function that issues a warning to the user or registered contacts when a potential fraud is detected.

[1226] "Means for monitoring the user's emotional state in real time and controlling the issuance of alerts in response to changes in the emotional state" refers to an emotion engine that analyzes the user's tone of voice and choice of words, detects changes in emotion, and appropriately adjusts the issuance of alerts.

[1227] A "specialized terminal" is a hardware device designed to perform a specific function.

[1228] A "generative AI model" is an artificial intelligence model that is trained to perform a specific task using natural language processing and machine learning techniques.

[1229] A system for implementing this invention includes means for collecting audio, means for analyzing the collected audio to detect potentially fraudulent conversations, means for issuing an alert when potentially fraudulent conversations are detected, and means for monitoring a user's emotional state in real time and controlling the issuance of alerts in response to changes in the emotional state.

[1230] System configuration

[1231] 1. Means of collecting audio:

[1232] The smartphone's microphone is used to capture the user's voice in real time.

[1233] The audio data is converted to text using a speech recognition library (e.g., speech_recognition library).

[1234] 2. Methods for analyzing collected audio to detect potentially fraudulent conversations:

[1235] Generative AI models (e.g., fraud detection models using the transformers library) are used to analyze text data and identify potentially fraudulent phrases and patterns.

[1236] If possible fraud is detected, the relevant text data is flagged.

[1237] 3. How to alert you to potentially fraudulent conversations:

[1238] When potential fraud is detected, alert the user or their registered contacts via voice message (e.g., using the pyttsx3 library) or automated phone call (e.g., using the smtplib library).

[1239] The alert is a message informing you of a possible scam.

[1240] 4. A means for monitoring the user's emotional state in real time and controlling the generation of alerts according to changes in the emotional state:

[1241] Using an emotion engine (e.g., an emotion analysis model using the transformers library), we analyze the user's tone of voice and vocabulary to monitor emotional changes in real time.

[1242] If the user's emotions change from anger or fear to relief or joy, the alerts will stop being generated.

[1243] Specific examples

[1244] If a user is talking to a scammer on the phone and a possible scam is detected, a voice message will be played saying, "This may be a scam. Please be careful," and a warning email will be automatically sent to family members.

[1245] If the user is feeling angry or scared, the emotion engine will detect that and alert them as well.

[1246] Prompt Sentence Examples

[1247] "Could this conversation be a scam?"

[1248] "Analyze the user's emotional state from this text."

[1249] In this way, the fraud prevention alert system can monitor conversations and emotions in real time and provide appropriate alerts to keep users safe.

[1250] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1251] Step 1:

[1252] The device collects the user's voice in real time using the smartphone's microphone. The input is the user's voice, and the output is voice data. This voice data is converted into text using a speech recognition library (e.g., speech_recognition library).

[1253] Step 2:

[1254] The server inputs the collected text data into a generative AI model (e.g., a fraud detection model using the transformers library) to analyze potentially fraudulent phrases and patterns. The input is text data, and the output is a judgment result on whether or not there is a possibility of fraud. If there is a possibility of fraud, the corresponding text data is flagged.

[1255] Step 3:

[1256] If the server determines that there is a possibility of fraud, it generates a voice message (using the pyttsx3 library, for example) and issues a warning to the user through the terminal. The input is the result of the determination that there is a possibility of fraud, and the output is the warning voice message.

[1257] Step 4:

[1258] If the server determines that there is a possibility of fraud, it automatically sends a warning email to registered contacts (e.g., family members or the police). The input is the result of the determination that there is a possibility of fraud, and the output is the warning email. The smtplib library is used to send the email.

[1259] Step 5:

[1260] The server analyzes the user's tone of voice and phrasing using an emotion engine (e.g., an emotion analysis model using the transformers library) to monitor the user's emotional state in real time. The input is text data, and the output is a judgment of the user's emotional state.

[1261] Step 6:

[1262] The server stops issuing alerts when the user's emotional state changes from anger or fear to relief or joy. The input is the result of the emotional state determination, and the output is an instruction to stop issuing alerts.

[1263] In this way, the fraud prevention alert system can collect the user's voice, analyze it for possible fraud, issue warnings as needed, and control the alerts according to the user's emotional state.

[1264] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1265] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1266] Another example of generative AI is Gemini (internet search engine). <url: https: gemini.google.com ?hl="ja">) are mentioned.

[1267] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1268] [Third embodiment]

[1269] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1270] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[1271] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1272] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1273] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1274] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1275] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1276] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1277] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1278] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1279] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1280] Next, the specific processing by the specific processing unit 290 of the data processing device 12 will be described.

[1281] "Example 1"

[1282] In one embodiment of the present invention, voice is collected using a dedicated terminal. This terminal is installed in a place frequently used by the elderly, such as a living room or bedroom. The terminal constantly monitors the elderly's conversations and collects their voice.

[1283] "Example 2"

[1284] The collected audio is analyzed using chat generation AI. The chat generation AI uses natural language processing technology to convert the audio into text and determine whether the text contains conversations that could be fraudulent. For example, if the text contains a phrase such as "I want you to send me money," it is determined to be a potential scam.

[1285] "Example 3"

[1286] If the device detects a potentially fraudulent conversation, it will issue an alert. The alert can be issued in various forms, such as a voice message from the device or an automated call to registered contacts (e.g., family members or the police). This makes it possible to prevent fraud before it occurs.

[1287] The processing flow of each embodiment will be described below.

[1288] "Example 1"

[1289] Step 1: Install a dedicated device in a location frequently used by the elderly, such as the living room or bedroom.

[1290] Step 2: The device constantly monitors the elderly person's conversations and collects their audio.

[1291] "Example 2"

[1292] Step 1: The collected audio is analyzed using chat generation AI.

[1293] Step 2: The chat generation AI uses natural language processing technology to convert speech into text.

[1294] Step 3: Determine if the text contains potentially fraudulent conversations. For example, if it contains a phrase like "please send me money," it is deemed potentially fraudulent.

[1295] "Example 3"

[1296] Step 1: If the device detects a potentially fraudulent conversation, it will issue an alert.

[1297] Step 2: The alert can take various forms, such as a voice message from the device or an automated call to registered contacts (e.g., family members or the police).

[1298] Example 1

[1299] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1300] There is a need to reduce the risk of elderly people becoming victims of fraud and improve their safety in their daily lives. However, conventional systems do not adequately collect and analyze voice data, making it difficult to detect potentially fraudulent conversations in real time. Furthermore, there is a lack of a way for users to easily check the analysis results. This has prevented the realization of effective measures to ensure the safety of elderly people.

[1301] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1302] In this invention, the server includes means for collecting voice, means for analyzing the collected voice and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, means for transmitting the collected voice data to the server, means for the server to analyze the voice data, means for saving the analysis results, and means for the user to check the analysis results. This makes it possible to monitor elderly people's conversations in real time, quickly detect conversations that may be fraudulent, and allow the user to easily check the analysis results.

[1303] The "means for collecting voice" is a dedicated terminal equipped with a highly sensitive microphone, which is a device that constantly monitors the elderly person's conversations and collects voice data.

[1304] The "means of analyzing collected audio to detect potentially fraudulent conversations" refers to a system that uses a generative AI model to convert audio data into text, analyzes the content of that text, and detects potentially fraudulent conversations.

[1305] The "means for issuing an alert when a potentially fraudulent conversation is detected" is a notification system for issuing a warning to the user when a potentially fraudulent conversation is detected.

[1306] The "means for transmitting collected voice data to a server" refers to a function in which the terminal connects to a server via the Internet and encrypts and transmits the collected voice data.

[1307] "Means by which the server analyzes voice data" refers to the process by which the server converts the voice data it receives into text using a generative AI model and analyzes the content.

[1308] The "means for saving the analysis results" is a function that allows the server to save the analysis results in a database, allowing users to easily search for the information they need later.

[1309] The "means for users to check the analysis results" refers to a system that allows users to access the server through a dedicated web interface or mobile app and view the analysis results.

[1310] MODE FOR CARRYING OUT THE INVENTION

[1311] This invention is a system for monitoring elderly people's conversations and detecting potentially fraudulent conversations. The system includes a dedicated terminal, a server, a generative AI model, and a user interface.

[1312] 1. Hardware and Software Configuration

[1313] Terminal

[1314] The device is equipped with a highly sensitive microphone and is installed in places frequently used by elderly people, such as living rooms and bedrooms, and has the function of temporarily storing collected voice data in local storage.

[1315] server

[1316] The server receives voice data from the device via the internet, has a generative AI model (e.g., OpenAI's GPT-4) installed on it, converts the voice data into text, and analyzes its content.

[1317] Generative AI Models

[1318] The generative AI model includes speech recognition technology to convert voice data into text and natural language processing technology to analyze the content of the text, thereby detecting potentially fraudulent conversations.

[1319] User Interface

[1320] Users can access the server and check the analysis results through a dedicated web interface or mobile app. The interface is designed to be easy for users to use.

[1321] 2. Data processing and calculation

[1322] Audio data collection

[1323] The device constantly monitors the elderly person's conversations and collects voice data. For example, if an elderly person is talking to a friend, such as "The weather is nice today," the voice is collected in real time.

[1324] Sending audio data

[1325] The collected voice data is sent from the device to the server at regular intervals (for example, every hour). The data is encrypted before being sent.

[1326] Analysis of audio data

[1327] The server converts the received voice data into text using a generative AI model. This conversion is performed using speech recognition technology. The generative AI model then analyzes the content of the text and extracts important information and unusual patterns. For example, the text "The weather is nice today" is generated and its content is analyzed.

[1328] Saving analysis results

[1329] The server stores the analysis results in a database, including the original audio data, the converted text, and the analysis results, allowing users to easily search for the information they need later.

[1330] Checking the analysis results

[1331] Users can access the server through a dedicated web interface or mobile app and check the analysis results. For example, if a family member wants to check the content of an elderly person's recent conversations, they can log in to the web interface and view the analysis result, such as "The weather is nice today."

[1332] 3. Examples and prompts

[1333] Specific examples

[1334] An elderly person is having a conversation with a friend in the living room, and the content of this conversation is collected by a device and analyzed using a generative AI model.

[1335] Prompt Sentence Examples

[1336] "Collect audio data of an elderly person having a conversation with a friend in their living room, and use a generative AI model to convert the conversation into text."

[1337] The above is an embodiment of the present invention.

[1338] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1339] Step 1:

[1340] The device collects the audio.

[1341] Input: Elderly people's conversation

[1342] How it works: A highly sensitive microphone installed in the device collects the elderly person's conversation in real time. For example, if an elderly person says to a friend, "The weather is nice today," the voice is collected.

[1343] Output: Collected audio data

[1344] Step 2:

[1345] The device stores the audio data in local storage.

[1346] Input: Collected audio data

[1347] Specific operation: The collected voice data is temporarily stored in the local storage of the device.

[1348] Output: Audio data saved in local storage

[1349] Step 3:

[1350] The terminal transmits the voice data to the server.

[1351] Input: Audio data stored in local storage

[1352] Specific operation: At regular intervals (for example, every hour), the device connects to a server via the Internet and encrypts and transmits the collected voice data.

[1353] Output: Audio data sent to the server

[1354] Step 4:

[1355] The server analyzes the audio data.

[1356] Input: Audio data sent to the server

[1357] How it works: The server converts the received voice data into text using a generative AI model (e.g., OpenAI's GPT-4). This conversion uses speech recognition technology. The generative AI model then analyzes the content of the text to extract important information and unusual patterns. For example, the text "The weather is nice today" is generated, and its content is analyzed.

[1358] Output: Parsed text data and analysis results

[1359] Step 5:

[1360] The server stores the analysis results.

[1361] Input: Parsed text data and analysis results

[1362] Specific operation: The server saves the analysis results in a database. The saved data includes the original voice data, the converted text, and the analysis results, allowing users to easily search for the information they need later.

[1363] Output: Analysis results stored in a database

[1364] Step 6:

[1365] The user checks the analysis results.

[1366] Input: Analysis results stored in the database

[1367] Specific operation: Users access the server through a dedicated web interface or mobile app to check the analysis results. For example, if a family member wants to check the content of an elderly person's recent conversations, they can log in to the web interface and view the analysis result, such as "The weather is nice today."

[1368] Output: Analysis results confirmed by the user

[1369] (Application example 1)

[1370] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1371] There is a need for systems in elderly living spaces that can quickly detect abnormal voice patterns (such as cries for help or the sound of a fall) and take appropriate action. However, conventional systems have low accuracy in analyzing voice data, which can result in abnormalities being overlooked. In addition, there is a lack of means to quickly issue an alert when an abnormality is detected, which can result in delayed response in an emergency. To solve these issues, a system capable of highly accurate voice analysis and rapid alert is needed.

[1372] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1373] In this invention, the server includes means for collecting voice, means for analyzing the collected voice and detecting abnormal voice patterns, and means for issuing an alert when an abnormal voice pattern is detected. This makes it possible to quickly detect abnormal voice patterns in the elderly person's living space and take appropriate measures.

[1374] The "means for collecting audio" is a dedicated terminal that is installed in the elderly person's living space and is used to constantly collect surrounding audio.

[1375] "Means for analyzing collected voice data and detecting abnormal voice patterns" refers to a function that analyzes collected voice data using a generative AI model to detect abnormal voice patterns such as calls for help or the sound of a fall.

[1376] "Means for issuing an alert when an abnormal voice pattern is detected" is a function that sends an alert to a device such as a smartphone when an abnormal voice pattern is detected, prompting a prompt response.

[1377] A "generative AI model" is an artificial intelligence model used to analyze voice data and detect anomalous voice patterns with high accuracy.

[1378] A "dedicated terminal" is a specialized device that is installed in the elderly person's living space and is used to collect voice.

[1379] The system for implementing this invention consists of a dedicated terminal installed in the elderly person's living space, a voice analysis system using a generative AI model, and a smartphone application that issues an alert when an abnormality is detected.

[1380] Hardware and Software Configuration

[1381] 1. Dedicated terminal: This device is installed in the elderly person's living space (living room, bedroom, etc.) and constantly collects surrounding sounds. This terminal is equipped with a microphone and a communication module for transmitting voice data to a server.

[1382] 2. Generative AI model: A server-based voice analysis system built using machine learning libraries such as TensorFlow and Keras, it analyzes collected voice data to detect abnormal voice patterns (such as calls for help or the sound of falling).

[1383] 3. Smartphone application: This application sends an alert to the user when an abnormal voice pattern is detected. When this application receives an alert, it notifies the user and prompts them to take action.

[1384] Processing flow

[1385] 1. Collection of voice data: A dedicated device collects voice data in the elderly person's living space and transmits it to a server in real time.

[1386] 2. Voice data analysis: A generative AI model on the server analyzes the collected voice data and detects anomalous voice patterns.

[1387] 3. Alerting: If an abnormal voice pattern is detected, the server sends an alert to the smartphone application, which displays a notification to the user, prompting them to take immediate action.

[1388] Specific examples

[1389] For example, if an elderly person falls in the middle of the night, a dedicated device will collect the sound and the generative AI model will recognize it as the "sound of a fall." If an abnormality is detected, the server will send an alert to the user's smartphone saying, "An elderly person has fallen."

[1390] Prompt Sentence Examples

[1391] "Voice data is collected from a dedicated device installed in the elderly's living space and analyzed using a generative AI model. If an abnormal voice pattern (such as a cry for help or the sound of a fall) is detected, an alert is sent to the smartphone."

[1392] In this way, it becomes possible to monitor the safety of elderly people in real time and respond quickly if an abnormality occurs.

[1393] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1394] Step 1:

[1395] The device collects audio in the elderly person's living space. The device uses a microphone to capture ambient audio in real time and converts the data into a digital format. The input is ambient audio, and the output is digital audio data.

[1396] Step 2:

[1397] The terminal transmits the collected digital voice data to the server. The terminal uses a communication module to transmit the collected voice data to the server via the Internet. The input is the digital voice data, and the output is the voice data transmitted to the server.

[1398] Step 3:

[1399] The server inputs the received voice data into a generative AI model. The server uses a generative AI model built using TensorFlow and Keras to analyze the voice data. The input is the voice data sent to the server, and the output is the analysis result.

[1400] Step 4:

[1401] The server detects abnormal voice patterns based on the analysis results of the generative AI model. The server evaluates the analysis results and determines whether they contain abnormal voice patterns such as calls for help or the sound of a fall. The input is the analysis results of the generative AI model, and the output is the anomaly detection results.

[1402] Step 5:

[1403] If the server detects an abnormal voice pattern, it sends an alert to the smartphone application. The server generates an alert message based on the anomaly detection results and sends it to the smartphone. The input is the anomaly detection result, and the output is the alert message.

[1404] Step 6:

[1405] A user receives an alert on a smartphone application. The application notifies the user of the received alert message and prompts them to take action. The input is the alert message, and the output is a notification to the user.

[1406] Example 2

[1407] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1408] Conventional voice analysis systems have difficulty detecting potentially fraudulent conversations in real time, putting users at high risk of becoming victims of fraud. Furthermore, the accuracy of voice data analysis is low, resulting in frequent false positives and false negatives. This creates a need for a highly reliable system to ensure user safety.

[1409] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1410] In this invention, the server includes means for receiving and storing voice data, means for converting the voice data into text, and means for analyzing the text to detect conversations that may be fraudulent, thereby enabling highly accurate analysis of voice data and real-time detection of conversations that may be fraudulent.

[1411] "Audio collection means" means a device or function for recording a user's speech and converting it into digital form.

[1412] The "means for transmitting collected voice to a server" is a device or function for transmitting voice data recorded by the terminal to a server via the Internet.

[1413] The "means for the server to receive and store voice data" refers to a device or function that allows the server to receive voice data sent from the terminal and store it in a specific folder or database.

[1414] The "means for the server to convert voice data into text" refers to a device or function for converting voice data received by the server into text format using natural language processing technology.

[1415] "Means for the server to analyze text and detect potentially fraudulent conversations" refers to a device or function that allows the server to analyze text data and detect specific phrases or patterns to determine whether there is a possibility of fraud.

[1416] "Means for issuing an alert when a potentially fraudulent conversation is detected" refers to a device or function for sending a warning message or notification to a user when the server detects a potentially fraudulent conversation.

[1417] A "dedicated terminal" is an electronic device specialized for collecting and transmitting audio to a server.

[1418] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to convert voice data into text and analyze the text data.

[1419] MODE FOR CARRYING OUT THE INVENTION

[1420] The present invention relates to a system for collecting voice data and detecting potentially fraudulent conversations. A specific embodiment of this system will be described below.

[1421] Audio data collection

[1422] The user uses the terminal to collect voice data. The terminal is equipped with a microphone that records the user's voice. For example, when the user says, "I want to send money," the voice is recorded on the terminal.

[1423] Sending audio data

[1424] The device transmits the collected voice data to a server. This requires an internet connection. Specifically, the device converts the voice data into a digital format and sends it to the server using an HTTP request.

[1425] Receiving and storing audio data

[1426] The server receives the voice data sent from the device. The server temporarily stores the received voice data and prepares it for analysis. For example, the server stores the voice data in a specific folder.

[1427] Converting audio data to text

[1428] The server passes the saved voice data to the chat generation AI, which uses natural language processing technology such as OpenAI's GPT-4. The server inputs the voice data into the AI, which then converts the voice into text.

[1429] Text Analysis

[1430] The server analyzes the converted text to determine whether it is potentially fraudulent. Specifically, the server searches for specific phrases in the text (e.g., "Please send me money") and evaluates whether it is potentially fraudulent.

[1431] Notification of results

[1432] The server sends the analysis results to the device, where the user can check the results. For example, the server sends a message to the device saying "Possible fraud," and the device displays the message to the user.

[1433] Specific examples

[1434] Example 1: Possible fraud

[1435] The user records a voice message saying "I want to send money," and the device sends the voice message to the server. The server converts the voice message into text and determines that it may be fraudulent. As a result, the device displays a message saying, "This may be fraudulent."

[1436] Example 2: No possibility of fraud

[1437] The user records "Hello, how are you?" and the device sends the audio to the server. The server converts the audio into text and determines that there is no possibility of fraud. As a result, the device displays a message saying "No possibility of fraud."

[1438] Prompt Sentence Examples

[1439] Here are some example prompts to input to a generative AI model:

[1440] "Convert the audio data into text and determine whether it is likely to be fraudulent. For example, if it contains a phrase like 'I want you to send me money,' determine that it is likely to be fraudulent."

[1441] Using this prompt, the generative AI model can properly analyze the voice data and determine the likelihood of fraud.

[1442] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1443] Step 1:

[1444] A user uses a terminal to collect voice data. The user presses the record button on the terminal and says, "I want to send money." The terminal records the voice and converts it into a digital format. The input is the user's voice, and the output is digital voice data.

[1445] Step 2:

[1446] The device sends the collected voice data to the server. The device encodes the voice data in a digital format and sends it to the server using an HTTP request. The input is the digital voice data, and the output is the voice data sent to the server.

[1447] Step 3:

[1448] The server receives and saves the audio data sent from the device. The server receives an HTTP request and saves the audio data in a specific folder. The input is the audio data sent from the device, and the output is the audio data saved on the server.

[1449] Step 4:

[1450] The server passes the voice data it has saved to the chat generation AI. The server inputs the voice data into the AI, which then converts the voice into text. The input is the voice data saved on the server, and the output is text data.

[1451] Step 5:

[1452] The server analyzes the converted text and determines whether it is likely to be fraudulent. The server searches for specific phrases in the text (e.g., "Please send me money") and evaluates whether it is likely to be fraudulent. The input is text data, and the output is the evaluation result regarding the likelihood of fraud.

[1453] Step 6:

[1454] The server sends the analysis results to the terminal. The server sends a message saying "Possible fraud" to the terminal, and the terminal displays the message to the user. The input is the evaluation result regarding the possibility of fraud, and the output is a warning message displayed to the user.

[1455] (Application example 2)

[1456] Next, a description will be given of Application Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1457] In recent years, fraud has become more sophisticated, with voice fraud in particular on the rise. This has increased the risk of many people becoming victims of fraud. Conventional fraud detection systems have the challenge of analyzing voice in real time and quickly determining whether a fraud is likely. Furthermore, there is a need for a system that can monitor the voice calls received by users and surrounding conversations in real time, and immediately issue a warning if there is a possibility of fraud.

[1458] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting voice, means for analyzing the collected voice and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, means for converting the collected voice into text, means for analyzing the converted text using a generative AI model, and means for inputting a prompt sentence into the generative AI model to determine the possibility of fraud. This makes it possible to monitor voice calls received by the user and surrounding conversations in real time and to immediately issue an alert if there is a possibility of fraud.

[1459] "Means for collecting audio" refers to devices or software for acquiring audio as digital data.

[1460] "Means for analyzing collected audio and detecting potentially fraudulent conversations" refers to devices or software that analyze acquired audio data and determine whether its content constitutes fraudulent activity.

[1461] "Means for issuing an alert when a potentially fraudulent conversation is detected" refers to a device or software that issues a warning to the user when a potentially fraudulent conversation is detected.

[1462] "Means for converting collected voice into text" refers to devices or software for converting voice data into text data.

[1463] "Means for analyzing converted text using a generative AI model" refers to devices or software that analyze text data using a generative AI model and understand its content.

[1464] "Means for inputting prompt sentences into a generative AI model to determine the possibility of fraud" refers to devices or software that input specific prompt sentences into a generative AI model and determine the possibility of fraud based on the response.

[1465] A system for implementing this invention includes means for collecting audio, means for analyzing the collected audio to detect potentially fraudulent conversations, means for issuing an alert when potentially fraudulent conversations are detected, means for converting the collected audio into text, means for analyzing the converted text using a generative AI model, and means for inputting a prompt sentence into the generative AI model to determine the likelihood of fraud.

[1466] Hardware and Software Configuration

[1467] 1. Audio collection method:

[1468] Hardware: Using the microphone in your smartphone or smart glasses.

[1469] Software: For audio collection, we use the speech_recognition library.

[1470] 2. Speech-to-text methods:

[1471] Hardware: Server or cloud service.

[1472] Software: Uses the Google Cloud Speech-to-Text API to convert collected audio into text in real time.

[1473] 3. Analysis methods using generative AI models:

[1474] Hardware: Server or cloud service.

[1475] Software: OpenAI's GPT-4 is used to analyze the converted text.

[1476] 4. Enter prompt text to determine potential fraud:

[1477] Hardware: Server or cloud service.

[1478] Software: Using OpenAI's API, specific prompts are fed into a generative AI model, which then determines the likelihood of fraud based on the response.

[1479] 5. Alert generation method:

[1480] Hardware: Smartphones and smart glasses.

[1481] Software: Applications that notify users of potential fraud.

[1482] Specific examples

[1483] For example, if a user receives a call containing the phrase "send me money," the system works as follows:

[1484] 1. Audio collection: The smartphone microphone collects audio.

[1485] 2. Speech-to-text conversion: The collected audio is converted to text using the Google Cloud Speech-to-Text API.

[1486] 3. Analysis using a generative AI model: The converted text is analyzed using OpenAI's GPT-4.

[1487] 4. Enter the prompt: Enter the following prompt into the generative AI model:

[1488] "Please determine if the following text is a potential scam: Send me money."

[1489] 5. Determining the likelihood of fraud: Determine the likelihood of fraud based on the response of the generative AI model.

[1490] 6. Alert: If a potential fraud attempt is detected, a warning notification will be sent to your smartphone.

[1491] The system allows users to detect potentially fraudulent conversations in real time and respond quickly.

[1492] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1493] Step 1:

[1494] A user collects audio using a smartphone or smart glasses. The input is ambient sound or phone conversation audio, and the output is digital audio data. Specifically, the device's microphone captures the audio and stores it as audio data.

[1495] Step 2:

[1496] The terminal sends the collected voice data to the server. The input is digital voice data, and the output is the voice data sent to the server. Specifically, the terminal's communication module uploads the voice data to the server via the Internet.

[1497] Step 3:

[1498] The server converts the voice data into text using the Google Cloud Speech-to-Text API. The input is digital voice data, and the output is text data. Specifically, the server sends the voice data to the API and receives the returned text data.

[1499] Step 4:

[1500] The server analyzes the converted text data using a generative AI model (OpenAI's GPT-4). The input is text data, and the output is the analysis result. Specifically, the server inputs the text data into the generative AI model and obtains the analysis result.

[1501] Step 5:

[1502] The server inputs a prompt sentence into the generative AI model to determine the likelihood of fraud. The input is the prompt sentence and text data, and the output is the judgment result regarding the likelihood of fraud. Specifically, the server inputs the following prompt sentence into the generative AI model:

[1503] "Please determine if the following text is a potential scam: Send me money."

[1504] Step 6:

[1505] If the server determines that there is a possibility of fraud, it issues an alert to the terminal. The input is the result of the judgment regarding the possibility of fraud, and the output is a warning notification to the user. Specifically, the server sends a notification to the terminal based on the judgment result, and the terminal displays a warning to the user.

[1506] Step 7:

[1507] The user receives the warning and takes the necessary action. The input is the warning notification, and the output is the user's response action. Specifically, the user checks the warning and takes appropriate action against the potentially fraudulent conversation.

[1508] Example 3

[1509] Next, a third embodiment of the third embodiment will be described. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1510] In recent years, fraud has become more sophisticated, with an increase in fraudulent activities using telephone and voice communications. Such fraudulent activities can cause significant damage, especially to elderly people and those unfamiliar with technology. Conventional methods have made it difficult to detect fraud and prevent damage before it occurs. Therefore, there is a need for a system that can detect potentially fraudulent conversations in real time and issue a prompt alert.

[1511] The specific processing by the specific processing unit 290 of the data processing device 12 in the third embodiment is realized by the following means.

[1512] In this invention, the server includes means for collecting voice, means for converting the collected voice into text data using voice recognition technology, means for analyzing the converted text data using a natural language processing model to detect conversations that may be fraudulent, and means for issuing an alert when a conversation that may be fraudulent is detected. This makes it possible to detect conversations that may be fraudulent in real time and to issue an alert quickly.

[1513] The "means for collecting voice" refers to a device or function for acquiring the user's conversation or voice in real time.

[1514] "Speech recognition technology" is a technology for converting acquired voice data into text data, allowing voice to be treated as text information.

[1515] "Text data" is character information converted using speech recognition technology, and is data in a format that can be analyzed and processed.

[1516] A "natural language processing model" is an artificial intelligence model that analyzes text data and determines whether it is likely to be fraudulent.

[1517] A "potentially fraudulent conversation" is a conversation that, after analysis by a natural language processing model, is determined to contain signs of fraudulent activity.

[1518] An "alert issuing means" is a device or function that issues a warning or notification to a user or a third party when a potentially fraudulent conversation is detected.

[1519] A "dedicated device" is a device specialized for collecting audio, including, for example, a smartphone or smart speaker.

[1520] A "generative AI model" is an artificial intelligence model that is used to analyze text data and determine the likelihood of fraud, and is trained for a specific task.

[1521] The present invention relates to a system that detects potentially fraudulent conversations in real time and issues an alert quickly. A specific embodiment of this system will be described below.

[1522] First, the device is equipped with a voice collection means for capturing the user's conversation in real time. The device is equipped with a microphone and records what the user says. Specific hardware used for this is a smartphone or smart speaker.

[1523] The device then converts the captured voice data into text using voice recognition technology. This process uses voice recognition software such as the Google Speech-to-Text API. Once the voice data is converted into text, it can be handled as text information.

[1524] The device then inputs the converted text data into a natural language processing model, specifically a generative AI model such as OpenAI's GPT-4, to analyze the text data and determine whether it is likely to be fraudulent. The model is trained to detect keywords and contexts related to fraud.

[1525] If the device determines that there is a high possibility of fraud based on the analysis results of the NLP model, it sets a fraud detection flag, which triggers the subsequent alert sending process.

[1526] Finally, the device will issue an alert if a fraud detection flag is raised. The alert can take multiple forms, including a voice message from the device or an automated call to registered contacts (e.g., family members or the police). Specifically, an automated call can be made using the Twilio API.

[1527] As a concrete example, consider the case where a user receives a phone call and is told, "Your bank account has been fraudulently used. Please tell us your account number and PIN to verify."

[1528] Process flow:

[1529] 1. The device picks up the user's conversation using a microphone. For example, the microphone on a smartphone records the user's voice.

[1530] 2. The device uses the Google Speech-to-Text API to convert the speech to text, which reads, "Your bank account has been compromised. Please provide your account number and PIN to verify."

[1531] 3. The device inputs the converted text into GPT-4 to analyze the possibility of fraud. GPT-4 detects keywords such as "bank account," "fraudulent use," "account number," and "PIN number," and determines that there is a high possibility of fraud.

[1532] 4. The device determines that there is a high possibility of fraud and sets a fraud detection flag.

[1533] 5. The device automatically calls the user's family using the Twilio API and notifies them that a potentially fraudulent conversation has been detected.

[1534] Example prompts to input to a generative AI model:

[1535] "When a user calls and says, 'Your bank account has been compromised. Please provide your account number and PIN to verify,' analyze whether this is a potential fraud."

[1536] This system makes it possible to prevent fraudulent acts before they occur. The flow of the identification process in the third embodiment will be described with reference to FIG.

[1537] Step 1:

[1538] The device captures the user's conversation in real time. Specifically, it uses a microphone installed on the device to collect voice data. The input is the user's voice, and the output is digital voice data. This voice data is used in subsequent processing steps.

[1539] Step 2:

[1540] The device converts the collected voice data into text data using voice recognition technology. Specifically, it uses the Google Speech-to-Text API to analyze the voice data and generate corresponding text data. The input is voice data and the output is text data. This text data is used in subsequent natural language processing steps.

[1541] Step 3:

[1542] The device inputs the converted text data into a natural language processing model. Specifically, it uses a generative AI model (e.g., OpenAI's GPT-4) to analyze the text data and determine whether it is likely to be fraud. The input is the text data, and the output is an analysis result indicating the likelihood of fraud. This analysis result is used to raise a fraud detection flag.

[1543] Step 4:

[1544] If the device determines that there is a high possibility of fraud based on the analysis results of the NLP model, it sets a fraud detection flag. The input is the analysis results of the NLP model, and the output is a fraud detection flag. This flag triggers the subsequent alert sending process.

[1545] Step 5:

[1546] If a fraud detection flag is raised, the device will issue an alert. Specifically, this can be a voice message from the device or an automated call to registered contacts (e.g., family members or the police). The input is the fraud detection flag and the output is an alert notification. Specific actions include making an automated call using the Twilio API.

[1547] (Application example 3)

[1548] Next, a description will be given of Application Example 3 of Form Example 3. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1549] In recent years, fraud has been on the rise, particularly over the phone. This has resulted in many people suffering financial losses. Conventional methods have made it difficult to detect potentially fraudulent conversations in real time and respond quickly. Therefore, there is a need for a system that can detect potentially fraudulent conversations in real time, quickly issue alerts, and, if necessary, automatically communicate with registered contacts.

[1550] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 3 is realized by the following means.

[1551] In this invention, the server includes means for collecting audio, means for analyzing the collected audio and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, and means for automatically communicating with registered contacts when a conversation that may be fraudulent is detected. This makes it possible to detect conversations that may be fraudulent in real time and respond quickly.

[1552] "Means for collecting audio" refers to devices and software for acquiring audio data.

[1553] "Means for analyzing collected audio and detecting potentially fraudulent conversations" refers to devices or software that analyze acquired audio data and identify content that may be fraudulent.

[1554] An "alert mechanism" is a device or software that issues a warning when it detects a potentially fraudulent conversation.

[1555] "Means for automatically communicating with registered contacts" refers to devices or software that automatically communicate with pre-registered contacts when a conversation that may be fraudulent is detected.

[1556] A "purpose-built terminal" is an electronic device designed for a specific purpose.

[1557] A "generative AI model" is a model generated using artificial intelligence techniques and trained to perform a specific task.

[1558] A system for implementing this invention includes means for collecting audio, means for analyzing the collected audio and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, and means for automatically communicating with registered contacts when a conversation that may be fraudulent is detected. Specific embodiments are described below.

[1559] Hardware and Software Configuration

[1560] The system is implemented using mobile devices such as smartphones. It uses the smartphone's microphone to collect audio. It uses the Python programming language and the SpeechRecognition library to analyze the audio data. It uses a generative AI model to detect potentially fraudulent conversations. It uses the Twilio API to trigger voice messages and automated phone calls to issue alerts.

[1561] Data processing and calculation

[1562] The device first collects voice data through a microphone. The collected voice data is converted to text data using the SpeechRecognition library. The text data is then analyzed using a generative AI model to detect potentially fraudulent keywords and phrases. If a potentially fraudulent conversation is detected, the device uses the Twilio API to alert the user with a voice message and automatically make a call to a registered contact.

[1563] Specific examples

[1564] For example, if a user says, "Please tell me my bank account information," the system will detect the keyword "bank account," issue an alert, and automatically call a registered contact.

[1565] Prompt Sentence Examples

[1566] "Create an application that detects potentially fraudulent conversations and alerts users. Include functionality to capture voice data and trigger alerts if fraudulent keywords are detected."

[1567] In this way, the present invention is able to detect potentially fraudulent conversations in real time and respond quickly.

[1568] The flow of the specific processing in Application Example 3 will be described with reference to FIG.

[1569] Step 1:

[1570] The device collects voice data using the smartphone's microphone. When the user starts a call, the device continues to capture voice data in real time. The input is the user's voice, and the output is the collected voice data.

[1571] Step 2:

[1572] The device converts the collected voice data into text data using the SpeechRecognition library. To convert voice data into text data, it uses Google's speech recognition API. The input is voice data, and the output is the converted text data.

[1573] Step 3:

[1574] The device analyzes text data using a generative AI model to detect potentially fraudulent keywords and phrases. The generative AI model scans the text data using a pre-trained list of fraudulent keywords. The input is the text data, and the output is a judgment of whether it is potentially fraudulent.

[1575] Step 4:

[1576] If a potentially fraudulent conversation is detected, the device uses the Twilio API to alert the user with a voice message. The input is the result of the potentially fraudulent conversation, and the output is an audio warning to the user.

[1577] Step 5:

[1578] Furthermore, if a potentially fraudulent conversation is detected, the device will automatically call registered contacts using the Twilio API. The input is the result of the potentially fraudulent conversation, and the output is an automated call to the registered contact.

[1579] In this way, the device can detect potentially fraudulent conversations in real time and respond quickly.

[1580] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1581] "Example 1"

[1582] In one embodiment of the present invention, the system includes an emotion engine that recognizes the user's emotions. The emotion engine recognizes emotions from the user's voice and issues an alert based on the user's emotional state. For example, if the user expresses negative emotions such as anger or fear, the system determines that fraud is a possibility and issues an alert.

[1583] "Example 2"

[1584] In addition, the emotion engine strengthens the detection of potentially fraudulent conversations when the user's emotional state deteriorates above a certain threshold. For example, if the user shows strong anger or fear, the emotion engine will detect that emotional state and strengthen the detection of potentially fraudulent conversations. This allows for more accurate detection of situations in which the user is likely to be scammed.

[1585] "Example 3"

[1586] Furthermore, the emotion engine monitors the user's emotional state in real time and controls the issuance of alerts according to changes in the emotional state. For example, if the user's emotion changes from anger or fear to relief or joy, the emotion engine detects this change in emotional state and stops issuing alerts. This allows the system to provide appropriate alerts according to the user's emotional state.

[1587] The processing flow of each embodiment will be described below.

[1588] "Example 1"

[1589] Step 1: Collect the user's voice using a dedicated device.

[1590] Step 2: The collected voice data is sent to the emotion engine to recognize the user's emotions.

[1591] Step 3: Detect potentially fraudulent conversations based on the emotional state recognized by the emotion engine.

[1592] Step 4: If a potentially fraudulent conversation is detected, issue an alert.

[1593] "Example 2"

[1594] Step 1: Collect the user's voice using a dedicated device.

[1595] Step 2: The collected voice data is sent to the emotion engine to recognize the user's emotions.

[1596] Step 3: Enhance detection of potentially fraudulent conversations when the emotional state recognized by the emotion engine deteriorates above a certain threshold.

[1597] Step 4: If a potentially fraudulent conversation is detected, issue an alert.

[1598] "Example 3"

[1599] Step 1: Collect the user's voice using a dedicated device.

[1600] Step 2: The collected voice data is sent to the emotion engine to recognize the user's emotions.

[1601] Step 3: Monitor the emotional state recognized by the emotion engine in real time and control the issuance of alerts according to changes in the emotional state.

[1602] Step 4: If the user's emotional state changes to a positive state, such as relief or joy, stop issuing alerts.

[1603] Example 1

[1604] Next, a description will be given of Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1605] In order to reduce the risk of elderly people falling victim to fraud, it is important to detect signs of fraud early in everyday conversations and take appropriate action. However, conventional systems lack sufficient analysis of voice data and emotion recognition, making it difficult to accurately determine the possibility of fraud. In addition, there is a lack of means to evaluate the user's emotional state, which can lead to delayed response when the user displays negative emotions. This has led to the issue of increasing the risk of elderly people falling victim to fraud.

[1606] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1607] In this invention, the server includes means for collecting voice, means for analyzing the collected voice to detect conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, means for converting voice data into text data, means for recognizing emotions from the text data, and means for evaluating the user's emotional state based on the emotion recognition results. This makes it possible to accurately recognize emotions from voice data and detect the possibility of fraud early. Furthermore, by evaluating the user's emotional state, it is possible to respond quickly when negative emotions are displayed, thereby reducing the risk of elderly people being scammed.

[1608] "Audio collection means" refers to a device or equipment for capturing ambient sounds, particularly a dedicated terminal that includes a highly sensitive microphone.

[1609] "Means for analyzing collected audio to detect potentially fraudulent conversations" refers to algorithms or software that analyze voice data and identify signs of fraud from its content.

[1610] "Means for issuing alerts when potentially fraudulent conversations are detected" refers to systems or features that send notifications to designated contacts when a potential fraudulent conversation is detected.

[1611] "Means for converting voice data into text data" refers to software or algorithms for converting voice data into text information using voice recognition technology.

[1612] "Means for recognizing emotions from text data" refers to emotion analysis algorithms or software that analyze text data and identify the user's emotions from its content.

[1613] "Means for evaluating the user's emotional state based on the emotion recognition results" refers to a system or function that comprehensively evaluates the user's emotional state based on the results of emotion analysis and takes appropriate measures as necessary.

[1614] This invention is a system for reducing the risk of elderly people falling victim to fraud, and by collecting and analyzing voice and recognizing emotions, it can detect the possibility of fraud early and issue an alert. A specific embodiment of this system will be described below.

[1615] Audio Collection

[1616] The device is installed in a place frequently used by the elderly, such as the living room or bedroom. The device is equipped with a highly sensitive microphone that constantly collects surrounding sounds. For example, the device can capture the elderly's voice saying, "Hello, how was your day?"

[1617] Sending audio data

[1618] The device transmits the collected audio data to the server in real time. The audio data is encrypted and transmitted using a secure communication protocol (e.g., HTTPS). For example, the device encrypts the audio data it captures and transmits it to the server using the HTTPS protocol.

[1619] Analysis of audio data

[1620] The server uses speech recognition software (e.g., Google Cloud Speech-to-Text API) to analyze the received voice data. The server converts the voice data into text. For example, the voice data "Hello, how was your day?" is converted into text data "Hello, how was your day?"

[1621] emotion recognition

[1622] The server inputs the text data into an emotion analysis algorithm (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. The emotion analysis algorithm identifies the type of emotion (e.g., anger, fear, joy, etc.) from the text data. For example, the phrase "Hello, how was your day?" is analyzed from the text data, and the emotion is recognized as "neutral."

[1623] Alert sending

[1624] The server evaluates the user's emotional state based on the results of emotion analysis. If the user shows negative emotions such as anger or fear, the server determines that there is a possibility of fraud and issues an alert. The alert is sent to designated contacts (e.g., family members or caregivers) via email or SMS. For example, if the user utters the text data "Help me, I'm scared!", the server recognizes the emotion as "fear" and determines that there is a possibility of fraud. The server then sends an email alert to the family saying, "Mom is scared. Please check on her."

[1625] Prompt Sentence Examples

[1626] An example of a prompt to be input to the generative AI model is as follows:

[1627] "Describe a system that collects audio data from elderly people conversing in their living rooms and recognizes emotions from the audio. Include the ability to issue an alert if the user expresses anger or fear."

[1628] Using this prompt, the generative AI model can generate a detailed description of the system.

[1629] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1630] Step 1:

[1631] The device is installed in a place frequently used by the elderly, such as the living room or bedroom. The device uses a highly sensitive microphone to constantly collect surrounding sounds. The input is the surrounding sounds, and the output is the collected voice data. For example, the device captures the elderly person's voice saying, "Hello, how was your day?"

[1632] Step 2:

[1633] The device transmits the collected audio data to the server in real time. The audio data is encrypted and transmitted using a secure communication protocol (e.g., HTTPS). The input is the collected audio data, and the output is the encrypted audio data. For example, the device encrypts the audio data it captures and transmits it to the server using the HTTPS protocol.

[1634] Step 3:

[1635] The server uses speech recognition software (e.g., Google Cloud Speech-to-Text API) to analyze the received voice data. The server converts the voice data into text. The input is encrypted voice data, and the output is text data. For example, the voice data "Hello, how was your day?" is converted into text data "Hello, how was your day?"

[1636] Step 4:

[1637] The server inputs the text data into an emotion analysis algorithm (e.g., IBM Watson Tone Analyzer) to recognize the user's emotion. The emotion analysis algorithm identifies the type of emotion (e.g., anger, fear, joy, etc.) from the text data. The input is the text data, and the output is the emotion recognition result. For example, the phrase "Hello, how was your day?" is analyzed from the text data, and the emotion is recognized as "neutral."

[1638] Step 5:

[1639] The server evaluates the user's emotional state based on the results of emotion analysis. If the user shows negative emotions such as anger or fear, the server determines that there is a possibility of fraud and issues an alert. The input is the emotion recognition result, and the output is an alert notification. For example, if the user utters text data such as "Help, I'm scared!", the server recognizes the emotion as "fear" and determines that there is a possibility of fraud. The server then sends an email alert to the family saying, "Mom is scared. Please check on her."

[1640] (Application example 1)

[1641] Next, a description will be given of Application Example 1 of Form Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1642] In order to reduce the risk of elderly people becoming victims of fraud, it is necessary to detect potentially fraudulent conversations in real time and respond quickly. However, conventional systems have issues with not being able to completely prevent fraud because they are unable to collect and analyze voice data properly and are unable to send alerts that take into account changes in emotions.

[1643] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 1 is realized by the following means. In this invention, the server includes means for collecting voice, means for analyzing the collected voice and detecting conversations that may be fraudulent, means for issuing an alert when a conversation that may be fraudulent is detected, means for recognizing emotions, and means for issuing an alert when the recognized emotions are negative. This reduces the risk of elderly people becoming victims of fraud and enables prompt response.

[1644] "Means for collecting voice" refers to devices or software for collecting the user's voice in real time.

[1645] "Means for analyzing collected audio to detect potentially fraudulent conversations" refers to algorithms or programs that analyze collected audio data and identify potentially fraudulent conversations from their content.

[1646] "Means for issuing an alert when a potentially fraudulent conversation is detected" refers to a device or software that issues a warning to a user or a third party when a potentially fraudulent conversation is detected.

[1647] "Means for recognizing emotions" refers to algorithms or programs that analyze a user's emotions from voice data and recognize a specific emotional state.

[1648] "Means for issuing an alert when the recognized emotion is negative" refers to a device or software for issuing a warning to a user or a third party when the recognized emotion is a negative state such as anger or fear.

[1649] "Specialized Device" means a hardware device designed and manufactured for a specific purpose.

[1650] A "generative AI model" is a model generated using artificial intelligence techniques and refers to an algorithm trained to perform a specific task.

[1651] This invention is a system for reducing the risk of elderly people becoming victims of fraud, and it has the ability to collect voice, recognize emotions, and detect potentially fraudulent conversations. The system is realized using a dedicated device, an emotion recognition engine, and a generative AI model.

[1652] System configuration

[1653] The system consists of the following main components:

[1654] 1. Dedicated terminal: A hardware device that is installed in a location frequently used by elderly people to collect their voice. This terminal has a built-in microphone and collects voice in real time.

[1655] 2. Emotion Recognition Engine: Software that analyzes user emotions from collected voice data and recognizes specific emotional states. This engine takes voice data as input and outputs an emotional state.

[1656] 3. Generative AI model: An algorithm that analyzes collected voice data and detects potentially fraudulent conversations. This model takes voice data as input and determines whether it is likely to be fraudulent.

[1657] 4. Alerting system: Software that warns users and third parties when potentially fraudulent conversations or negative sentiment are detected. Alerts are sent via email and smartphone notifications.

[1658] Program processing explanation

[1659] The server collects voice data from a dedicated device and analyzes it using an emotion recognition engine and generative AI model. Specifically, the following process is carried out:

[1660] 1. Voice collection: A dedicated device collects the voice of the elderly person in real time. The collected voice data is sent to a server.

[1661] 2. Emotion Recognition: The server uses an emotion recognition engine to analyze the user's emotions from the voice data. For example, if the user expresses anger or fear, the emotional state is recognized.

[1662] 3. Fraud Detection: The server analyzes the audio data using a generative AI model to detect potentially fraudulent conversations. If fraud is detected, the information is sent to an alert system.

[1663] 4. Alerting: When potentially fraudulent conversations or negative sentiment are detected, the alerting system will warn users or third parties via email or smartphone notifications.

[1664] Specific examples

[1665] For example, if an elderly person is talking to a scammer on the phone, and emotions such as "anger" or "fear" are detected in the conversation, an alert email will be sent to the family stating, "An elderly person is showing signs of anger. There is a possibility of fraud."

[1666] Prompt Sentence Examples

[1667] "Develop a system that collects voices from elderly people and recognizes their emotions. If emotions such as anger or fear are detected in the voice, the application will send an alert email to their family members. The libraries used are speech_recognition, emotion_recognition, and smtplib."

[1668] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1669] Step 1:

[1670] A dedicated terminal collects the elderly person's voice in real time. The terminal's microphone captures the voice data and sends it to a server. The input is the elderly person's voice, and the output is the voice data sent to the server.

[1671] Step 2:

[1672] The server inputs the received voice data into an emotion recognition engine, which analyzes the voice data and identifies the user's emotional state. The input is the voice data, and the output is the recognized emotional state (e.g., anger, fear).

[1673] Step 3:

[1674] The server evaluates the emotional state obtained from the emotion recognition engine, and if a negative emotion (e.g., anger, fear) is detected, it sends the information to the alerting system. The input is the recognized emotional state, and the output is the emotional information sent to the alerting system.

[1675] Step 4:

[1676] The server inputs the voice data into a generative AI model, which analyzes the voice data and detects potentially fraudulent conversations. The input is the voice data, and the output is a judgment result on whether or not there is a possibility of fraud.

[1677] Step 5:

[1678] The server evaluates the judgment results obtained from the generative AI model, and if it determines that there is a possibility of fraud, it sends that information to the alert system. The input is the judgment result of whether there is a possibility of fraud, and the output is the fraud information sent to the alert system.

[1679] Step 6:

[1680] The alert system issues warnings to users and third parties based on emotional information and fraud information received from the server. The warnings are sent via email or smartphone notifications. The input is emotional information and fraud information, and the output is a warning message sent to users and third parties.

[1681] Example 2

[1682] Next, a description will be given of Example 2 of Form Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1683] While conventional voice analysis systems can detect potentially fraudulent conversations, they have the problem of making it difficult to accurately assess the risk of fraud because they do not take the user's emotional state into account. Furthermore, they are unable to adequately detect situations in which the likelihood of fraud increases when the user is expressing strong emotions, which makes it difficult to adequately ensure the user's safety.

[1684] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1685] In this invention, the server includes means for collecting audio, means for analyzing the collected audio to detect potentially fraudulent conversations, means for issuing an alert when potentially fraudulent conversations are detected, means for detecting the user's emotional state, and means for enhancing the detection of potentially fraudulent conversations when the user's emotional state deteriorates to a certain threshold or greater, thereby enabling accurate determination of fraud risk taking the user's emotional state into account.

[1686] A "means for collecting voice" is a device or software for acquiring voice data provided by a user in digital form and transmitting it to a server.

[1687] "Means for analyzing collected audio to detect potentially fraudulent conversations" refers to technology that converts audio data into text and analyzes that text to identify potentially fraudulent phrases and patterns.

[1688] The "means for issuing an alert when a potentially fraudulent conversation is detected" refers to a device or software for issuing a warning to a user when a potentially fraudulent conversation is detected.

[1689] "Means for detecting the user's emotional state" refers to technology for analyzing the user's voice and facial expression data to evaluate their emotional state.

[1690] "Means for strengthening the detection of potentially fraudulent conversations when a user's emotional state deteriorates above a certain threshold" refers to technology that more strictly detects potentially fraudulent conversations when a user is expressing strong emotions such as anger or fear.

[1691] A "generative AI model" is an artificial intelligence model that uses natural language processing technology to analyze text data and identify specific patterns and phrases.

[1692] A "prompt sentence" is text data that is input into a generative AI model and is a sentence that specifically indicates the content of the object to be analyzed.

[1693] The present invention relates to a system for collecting voice data and detecting potentially fraudulent conversations. A specific embodiment of this system will be described below.

[1694] Audio data collection

[1695] The user provides voice data. The device collects this voice using a built-in microphone and converts it into a digital format. The device then transmits the collected voice data to a server.

[1696] Converting audio data to text

[1697] The server receives the voice data sent from the device. The server calls the Google Cloud Speech-to-Text API and converts the voice data into text. For example, the text generated is "Hello, how are you doing lately?"

[1698] Detecting potential fraud

[1699] The server inputs the generated text into a generative AI model (e.g., OpenAI's GPT-4). The generative AI model uses natural language processing techniques to analyze whether the text is likely to be fraudulent. For example, if the text contains the phrase "please send me money," it will determine that it is likely to be fraudulent.

[1700] Emotional state detection

[1701] The device collects the user's voice and facial expression data and sends it to a server, which then uses Affectiva's emotion recognition API to analyze the user's emotional state. For example, if the user is expressing strong anger or fear, the server can detect that emotional state.

[1702] Enhanced fraud detection

[1703] The server will strengthen its detection of potentially fraudulent conversations if the user's emotional state deteriorates above a certain threshold. For example, if the user expresses strong fear when asked to "send money," it will determine that the conversation is likely to be fraudulent.

[1704] Notification of results

[1705] If the server determines that there is a possibility of fraud, it notifies the terminal of the result. The terminal displays a warning to the user, saying, "There is a possibility of fraud. Please be careful."

[1706] Specific examples

[1707] The user says, "I want you to send me money." The device collects this audio and sends it to the server. The server converts the audio into text and inputs it into a generative AI model. The generative AI model determines that there is a possibility of fraud. At the same time, the device collects the user's facial expression data and sends it to the server. The server uses Affectiva's emotion recognition API to detect that the user is showing strong fear. The server determines that there is a high possibility of fraud and notifies the device to display a warning. The device displays a message to the user saying, "This may be a scam. Please be careful."

[1708] Prompt Sentence Examples

[1709] "Please help us determine if the following text is a potential scam: 'I want you to send me money'"

[1710] This system is expected to protect users from the risk of fraud.

[1711] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1712] Step 1:

[1713] The user provides voice data. The device uses the built-in microphone to collect this voice and convert it into digital form. The input is the user's voice data, and the output is digital voice data. In concrete terms, the device captures voice through the microphone and performs digital signal processing.

[1714] Step 2:

[1715] The terminal transmits the collected digital voice data to the server. The input is the digital voice data, and the output is the voice data transmitted to the server. Specifically, the terminal uploads the voice data to the server via a network.

[1716] Step 3:

[1717] The server receives the voice data sent from the terminal. The input is the voice data sent from the terminal, and the output is the voice data stored in the server. In concrete terms, the server receives the voice data via the network and stores it in a database.

[1718] Step 4:

[1719] The server calls the Google Cloud Speech-to-Text API to convert the voice data to text. The input is the voice data stored on the server, and the output is text data. Specifically, the server calls the API and sends a request to convert the voice data to text.

[1720] Step 5:

[1721] The server inputs the generated text data into a generative AI model (for example, OpenAI's GPT-4) and analyzes whether there is a possibility of fraud. The input is text data, and the output is the analysis result regarding the possibility of fraud. Specifically, the server inputs the text data into the generative AI model and obtains the analysis result.

[1722] Step 6:

[1723] The device collects the user's voice and facial expression data and sends it to the server. The input is the user's voice and facial expression data, and the output is the emotion data sent to the server. Specifically, the device uses a camera and microphone to capture the user's facial expressions and voice, converts them into digital format, and sends them to the server.

[1724] Step 7:

[1725] The server uses Affectiva's emotion recognition API to analyze the user's emotional state. The input is the emotional data sent to the server, and the output is the analysis result of the user's emotional state. Specifically, t...

Claims

1. A means for collecting audio; A conversion means for converting the collected voice into text using natural language processing technology; A detection means for detecting whether the collected voice includes a conversation that may be fraudulent by inputting the converted text and a prompt sentence that instructs the model to determine whether the text includes a conversation that may be fraudulent into a generative AI model; A method of automatically sending alerts to registered contacts when a conversation containing a possible fraudulent message is detected; A system including:

2. The method further includes a recognition means for recognizing an emotional state of a user from the collected voice of the user using an emotion engine; the detection means enhances detection of potentially fraudulent conversations when the recognized emotional state of the user deteriorates above a certain threshold. The system of claim 1 .

3. The method further includes a monitoring means for monitoring an emotional state of the user in real time from the collected voice of the user using the emotion engine; The transmission means controls transmission of an alert in response to a change in the emotional state of the user. The system of claim 1 .

Images