System

Abstract

A system is provided.SOLUTION: A system comprising: means for converting voice data into text data in real time; means for analyzing the text data using a generative artificial intelligence model and evaluating a possibility of fraud or criminal behavior; and means for displaying a warning message and a countermeasure proposal to a user based on an evaluation result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] As IT services become more diverse and complex, fraud and criminal activity is on the rise. The damage caused by this is particularly pronounced among the elderly and young people with low IT literacy, making it a social problem. Furthermore, these criminal activities are not limited to online activities; they can also be seen in offline settings, such as street solicitations and door-to-door sales. There is a need for a system that can comprehensively solve these problems and protect users from fraud and crime. [Means for solving the problem]

[0005] To solve the above problems, the present invention provides the following means. A system is provided that includes means for converting voice data into text data in real time, means for analyzing the text data using a generative artificial intelligence model to evaluate the likelihood of fraud or criminal activity, and means for displaying a warning message and suggested countermeasures to the user based on the evaluation results. This system allows users to know in real time whether the content of a telephone or face-to-face conversation is likely to be fraud or criminal activity. Furthermore, by providing users with appropriate countermeasures, it is possible to prevent damage from occurring. Furthermore, long-term accuracy improvements can be achieved by securely communicating the text data required for analysis and by periodically retraining the generative artificial intelligence model.

[0006] "Audio data" refers to data in a format in which sound waves are recorded or transmitted as digital information.

[0007] "Text data" refers to digital information that includes characters and symbols, and includes audio data converted into characters.

[0008] "Real-time" refers to processing or communication that has little or no delay time.

[0009] "Conversion" is the process of changing data of one format into data of another format.

[0010] A "generative artificial intelligence model" is an AI technology that learns patterns and rules from data and makes predictions and classifications based on new data.

[0011] "Analysis" is the process of examining specific data and making judgments or conclusions based on it.

[0012] "Evaluation" is the process of analyzing data and determining its value, usefulness, risks, etc.

[0013] A "warning message" is a notification message that informs the user that a particular risk or problem exists.

[0014] "Countermeasure proposals" are specific advice that indicate actions or measures that users should take in response to specific risks or problems.

[0015] "User" refers to any person or entity that operates this system.

[0016] A "system" is a collection of devices and programs in which multiple components work together to provide a specific function.

[0017] "Secure" means that information is protected from unauthorized access and unauthorized modification.

[0018] "Encryption" is the process of transforming data based on a specific algorithm to prevent unauthorized reading.

[0019] "Communication" refers to sending and receiving data and is the process of exchanging information between different devices.

[0020] "Retraining" is the process of retraining an existing artificial intelligence model using new data to improve its performance.

[0021] A "pattern database" is a database that systematically accumulates characteristic patterns and cases related to fraud and criminal activity.

[0022] ---

[0023] These are the definitions for each of the key words in the patent claims, which will help clarify the meaning of each term in the patent specification. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0025] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0026] First, the terms used in the following description will be explained.

[0027] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0028] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0029] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0030] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0031] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0032] [First embodiment]

[0033] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0034] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0035] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0036] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0037] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0038] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0039] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0040] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0041] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0042] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0043] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0044] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0045] This invention is a system that converts voice data into text data in real time, evaluates the possibility of fraud or criminal activity using a generative artificial intelligence model, and presents a warning and countermeasure suggestions to the user. Specific embodiments will be described below.

[0046] System configuration and operation

[0047] 1. The user launches the app

[0048] Users launch a dedicated app on their smartphone or tablet.

[0049] The device will display the app interface and start the audio capture function.

[0050] When a user starts a phone or in-person conversation, the app automatically begins capturing audio.

[0051] 2. Real-time transcription of conversations

[0052] The device converts the captured audio into text data in real time, using voice recognition technology to instantly convert spoken words into written text.

[0053] Once the transcription is complete, the text data is sent to the server.

[0054] 3. Analysis and Evaluation of Text Data

[0055] The server inputs the received text data into a generative artificial intelligence model.

[0056] The generative AI model analyzes the content of the conversation by comparing it with a database of patterns of fraud and criminal activity.

[0057] If fraud or criminal activity is suspected, the server evaluates the results and generates a warning message.

[0058] 4. Present warnings and suggested solutions

[0059] The server returns data including a warning message and suggested countermeasures to the terminal.

[0060] The device will warn the user with a pop-up message and suggest appropriate measures, such as "This may be a scam" or "Please check with official contacts."

[0061] 5. Recording conversation patterns

[0062] The terminal sends the data of the completed conversation to the server, where it is recorded in the backend.

[0063] The server analyzes the stored data and uses it as data for retraining the generative AI model.

[0064] Specific examples

[0065] Scenario: Fake billing call

[0066] 1. Launch the app

[0067] An elderly user launches a dedicated app.

[0068] The user prepares to receive the call.

[0069] 2. Start a conversation and transcribe

[0070] The user receives a call and the conversation begins with the user being told that the outstanding amount is due and that they should pay it immediately.

[0071] The device captures the conversation in real time and instantly transcribes it into text.

[0072] 3. Sending and analyzing text data

[0073] The terminal transmits the generated text data to the server.

[0074] The data received by the server is analyzed using a generative AI model to detect fraudulent patterns such as "unpaid" and "pay immediately."

[0075] 4. Displaying warnings and suggested solutions

[0076] The server determines that there is a high possibility of fraud and generates a warning message and a suggestion to "check with the official contact point."

[0077] The terminal displays these messages to the user as popups.

[0078] 5. Recording conversation patterns

[0079] After the conversation is over, the device sends all data to the server.

[0080] The server records the data and uses it to retrain the generative AI model.

[0081] This system not only effectively protects users from fraud and criminal activity, but also allows the accuracy of the generative AI model to be constantly improved based on the collected data.The system covers both online and offline risks, making it effective for a wide range of users.

[0082] The processing flow will be explained below.

[0083] Step 1:

[0084] The user launches the dedicated app on their smartphone or tablet. The device displays the app interface and starts the audio capture function.

[0085] Step 2:

[0086] The user answers a phone call or starts a face-to-face conversation. The device begins capturing audio through the microphone. Once audio capture begins, the device transcribes the captured audio data in real time.

[0087] Step 3:

[0088] The device generates the transcribed text data and sends it to the server, where it is encrypted using a secure communication protocol.

[0089] Step 4:

[0090] The server inputs the received text data into a generative AI model, which then compares it with a database of fraud and criminal patterns and analyzes the conversation.

[0091] Step 5:

[0092] The generative AI model returns the analysis results to the server, which evaluates them and determines whether there is any potential for fraud or criminal activity.

[0093] Step 6:

[0094] The server generates a warning message based on the evaluation results, including specific warning content and suggested countermeasures for the user.

[0095] Step 7:

[0096] The server sends back data containing a warning message and suggested solutions to the terminal, which receives it and displays it to the user as a pop-up message.

[0097] Step 8:

[0098] The user ends the call. The device sends all conversation data (audio and transcription data) to the server.

[0099] Step 9:

[0100] The server records the data it receives and stores it in a back-end database, which can then be used to retrain the AI ​​model in the future.

[0101] Step 10:

[0102] The server periodically retrains the generative AI model based on new conversation patterns, thereby improving the model's detection accuracy.

[0103] ---

[0104] This is a detailed explanation of the program's processing steps, which will give you a clear understanding of how the system works and what specific processes are performed.

[0105] Example 1

[0106] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0107] Conventional methods have had difficulty detecting fraud and criminal activity in real time and quickly warning users. They also lacked the means to securely transmit voice data and properly return analysis results and warning messages to the device. Therefore, there was a need for an effective means for users to prevent themselves from becoming victims of fraud and criminal activity.

[0108] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0109] In this invention, the server includes means for converting voice data into text data in real time, means for analyzing the text data using a generative artificial intelligence model to evaluate the likelihood of fraud or criminal activity, means for displaying a warning message and suggested countermeasures to the user based on the evaluation results, means for transmitting text data transmitted from the user to the terminal to the server, means for analyzing the text data using the generative artificial intelligence model to detect the likelihood of fraud or criminal activity, means for returning the generated warning message and suggested countermeasures to the terminal, and means for recording the conversation data and using it as data for retraining, thereby enabling users to be effectively protected from fraud and criminal activity.

[0110] "Audio data" means data in digital form that contains human speech or other sound information.

[0111] "Text data" is data consisting of characters and symbols that is obtained as a result of converting voice data.

[0112] A "generative artificial intelligence model" is an artificial intelligence model with natural language processing capabilities that is trained based on a large dataset.

[0113] "Analysis" is the process performed to determine the possibility of fraud or criminal activity based on the input data.

[0114] A "warning message" is a message that is generated as a result of analysis to warn or alert the user.

[0115] "Measure suggestions" are specific actions or measures recommended to the user based on the warning message.

[0116] "User" refers to any individual or corporation that uses this system.

[0117] "Terminal" refers to an electronic device, such as a smartphone or tablet, that allows a user to access and use the system.

[0118] A "server" is a central computer system used to process voice and text data and provide information to users.

[0119] "Real-time" refers to processing that occurs almost simultaneously with the occurrence of an event.

[0120] "Conversation data" refers to data that includes captured speech and text data generated based on the speech.

[0121] "Retraining" is the process of retraining a generative artificial intelligence model using new data to improve its accuracy.

[0122] MODE FOR CARRYING OUT THE INVENTION

[0123] The present invention provides a system that converts voice data into text data in real time, evaluates the possibility of fraud or criminal activity using a generative artificial intelligence model, and presents a warning and countermeasure suggestions to the user. A specific embodiment of the present invention will be described below.

[0124] System configuration and operation

[0125] The user launches the app

[0126] Users launch a dedicated app on their smartphone or tablet. This app has a voice capture function, and when the user starts a phone or face-to-face conversation, the device begins capturing the voice in real time.

[0127] Voice capture to text conversion

[0128] The device converts the captured voice data into text data in real time using voice recognition software (e.g., Google Speech-to-Text API), and the converted text data is immediately sent to the server.

[0129] Text data analysis

[0130] The server inputs the received text data into a generative AI model (e.g., OpenAI GPT-4). The generative AI model analyzes the conversation by comparing it with a database of fraudulent and criminal patterns. If the analysis reveals a possibility of fraud or criminal activity, the server generates a warning message and suggests countermeasures.

[0131] Warnings and suggested solutions

[0132] The server generates a warning message and sends it back to the device, which displays it to the user as a pop-up message. Specific messages include "This may be a scam" and "Please check with your official contacts."

[0133] Recording and retraining conversation data

[0134] After the conversation is complete, the device sends all conversation data to the server, which records it in a database. This data is used to retrain the generative AI model, helping to improve the accuracy of the system.

[0135] Specific operation example

[0136] Scenario: Fake billing call

[0137] 1. Launch the app

[0138] An elderly user launches a dedicated app.

[0139] The user prepares to receive the call.

[0140] 2. Start a conversation and transcribe

[0141] The user receives a call and the conversation begins with the user being told that the outstanding amount is due and that they should pay it immediately.

[0142] The device captures the conversation in real time and instantly transcribes it into text.

[0143] 3. Sending and analyzing text data

[0144] The terminal transmits the generated text data to the server.

[0145] The data received by the server is analyzed using a generative artificial intelligence model to detect fraudulent patterns such as "non-payment" and "immediate payment."

[0146] 4. Displaying warnings and suggested solutions

[0147] The server determines that there is a high possibility of fraud and generates a warning message and a suggestion to "check with the official contact point."

[0148] The terminal displays these messages to the user as popups.

[0149] 5. Recording conversation patterns

[0150] After the conversation is over, the terminal sends all data to the server.

[0151] The server records the data and uses it to retrain the generative AI model.

[0152] In this way, the system of the present invention can consistently convert voice data in real time, analyze it using a generative AI model, and promptly warn users and suggest countermeasures. This system effectively protects users from fraud and criminal activity, and the accuracy of the system is continuously improved.

[0153] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0154] Step 1:

[0155] The user launches the app

[0156] Input: The user operates a smartphone or tablet device to launch a dedicated app.

[0157] What it does: The device displays the app interface and puts the audio capture feature into standby.

[0158] Output: The audio capture feature is enabled and audio is recorded when the user starts speaking.

[0159] Step 2:

[0160] The device captures audio in real time

[0161] Input: Ambient sounds, especially phone calls and face-to-face conversations you have.

[0162] What it does: Your device uses the app's audio capture feature to record audio data in real time.

[0163] Output: Recorded audio data.

[0164] Step 3:

[0165] The device converts the voice data into text data.

[0166] Input: Real-time recorded audio data.

[0167] How it works: Your device uses speech recognition software (e.g., Google Speech-to-Text API) to convert voice data into text.

[0168] Output: The converted text data.

[0169] Step 4:

[0170] The device sends text data to the server.

[0171] Input: The converted text data.

[0172] How it works: The device sends text data to the server using a secure communication protocol (e.g., HTTPS).

[0173] Output: Text data is sent to the server.

[0174] Step 5:

[0175] The server analyzes the text data using a generative AI model

[0176] Input: The text data sent to the server.

[0177] How it works: The server feeds text data into a generative artificial intelligence model (e.g., OpenAI GPT-4) to analyze it for potential fraud or criminal activity.

[0178] Output: Analysis results in assessment data of the likelihood of fraud or criminal activity.

[0179] Step 6:

[0180] The server evaluates the results and generates a warning message

[0181] Input: Analysis results generated by a generative AI model.

[0182] How it works: The server evaluates the analysis results and generates a warning message and suggested actions if there is a high possibility of fraud or criminal activity.

[0183] Output: Generated warning messages and suggested actions.

[0184] Step 7:

[0185] The server sends a warning message and suggested solutions back to the device.

[0186] Input: Generated warning message and suggested action.

[0187] Operation: The server returns a warning message and suggested solutions to the device as an HTTP response.

[0188] Output: A warning message and suggested solutions will be sent to your device.

[0189] Step 8:

[0190] The device displays a warning and suggests countermeasures to the user.

[0191] Input: The warning message and suggested action returned by the server.

[0192] How it works: The device displays a pop-up notification to the user with a warning message and suggested solutions.

[0193] Output: User confirms warning message and suggested action.

[0194] Step 9:

[0195] The device records conversation patterns and sends them to the server.

[0196] Input: Finished conversation data.

[0197] Operation: The terminal records the conversation data that has been completed and sends it to the server.

[0198] Output: Conversation data recorded on the server.

[0199] Step 10:

[0200] The server records the data and uses it to retrain the generative AI model.

[0201] Input: Conversation data sent to the server.

[0202] How it works: The server stores conversation data and uses it as retraining data for generative AI models.

[0203] Output: An updated generative AI model and conversation data recorded in a database.

[0204] (Application example 1)

[0205] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0206] In modern society, fraud and criminal acts are becoming increasingly sophisticated, and many people are falling victim to these methods. To address this situation, a system is needed that can detect fraud and criminal acts early and provide appropriate warnings and countermeasures. However, current technology does not provide a system that can analyze voice data in real time and immediately evaluate the possibility of fraud and provide a warning. Therefore, an objective of the present invention is to provide a system that can detect fraud and criminal acts in real time and provide a warning, allowing users to communicate safely.

[0207] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0208] In this invention, the server includes means for converting voice data into text data in real time, means for analyzing the text data using a generative artificial intelligence model to evaluate the possibility of fraud or criminal activity, means for displaying a warning message and countermeasure suggestions to the user based on the evaluation results, means for providing a voice capture function, means for converting voice into text using voice recognition technology, means for analyzing the text by matching it with fraud patterns, and means for sending a warning message to the user using communication means, thereby enabling the user to be aware of the possibility of fraud or criminal activity in real time and take appropriate countermeasures.

[0209] "Audio data" refers to continuous sound signals obtained from conversations, communications, etc.

[0210] "Text data" is voice data converted into a string of characters in a language.

[0211] A "generative artificial intelligence model" is an AI technology that learns large amounts of data and generates and analyzes sentences and information.

[0212] "Voice recognition technology" is a technology that converts voice data into text in real time.

[0213] "Fraud patterns" are data that indicate combinations of words and phrases associated with fraud.

[0214] A "warning message" is a notification that alerts the user to danger or problems.

[0215] "Measure suggestions" are advice that shows specific actions or methods that users should take.

[0216] "User" refers to any person or organization that uses this system.

[0217] The "audio capture function" is a function for collecting audio in real time.

[0218] "Communication means" refers to the technology and protocols used to exchange information.

[0219] A "server" is a computer system that processes and stores data and provides services over a network.

[0220] "Terminal" refers to a device or equipment that is directly operated by a user.

[0221] "Analysis results" refers to the results of analysis using a generative artificial intelligence model.

[0222] MODE FOR CARRYING OUT THE INVENTION

[0223] This invention is a security system for users that converts voice data into text data in real time, analyzes the text data using a generative artificial intelligence model to evaluate the possibility of fraud or criminal activity, and provides the user with a warning and suggestions for countermeasures. This system is mainly composed of the following elements.

[0224] 1. Hardware and Software Selection

[0225] The following hardware and software are used to realize this system.

[0226] Smartphone terminal: A device operated by a user.

[0227] Microphone: Used to capture audio data.

[0228] Server: Processes data and manages the database.

[0229] Google Cloud Speech-to-Text API: Provides speech recognition technology.

[0230] Generative AI models (e.g., OpenAI GPT): Analyze text data to assess the likelihood of fraud or criminal activity.

[0231] Twilio API: A communication method for sending alert messages to users.

[0232] 2. Explanation of program processing

[0233] The main processing flow of the program of this system is explained below.

[0234] User voice input

[0235] The user starts a conversation using a smartphone. The device's voice capture function is activated, capturing the conversation's audio data in real time. This allows the user's conversation to be input into the system in real time and prepared for analysis.

[0236] Converting audio data to text

[0237] The server converts the captured audio data into text data using the Google Cloud Speech-to-Text API, which then sends the audio data to the server as text data.

[0238] Text data analysis

[0239] The server uses a generative AI model (OpenAI GPT) to analyze the converted text data. This identifies patterns of fraud and criminal activity and assesses risk. This analysis process is performed by using specific keywords and phrases as prompts. The following are example prompts:

[0240] Rate the following text as a scam: Your credit card details may have been stolen. Check now

[0241] Warning messages and suggested solutions

[0242] If the server determines based on the analysis results that there is a high possibility of fraud or criminal activity, it uses the Twilio API to send a warning message to the user's device. The warning message also includes specific suggestions for countermeasures. For example, a message such as "This may be a scam. Please contact an official contact point" may be displayed.

[0243] Adding specific examples

[0244] As a concrete example, let's consider a scenario in which an elderly user receives a fraudulent phone call. When the user receives a call stating, "You have an outstanding payment and need to pay it immediately," the audio is captured in real time and converted to text using the Google Cloud Speech-to-Text API. The OpenAI GPT model then evaluates the likelihood of fraud and, if it determines there is a high risk, sends a warning message to the user's smartphone via the Twilio API stating, "There is a high possibility of fraud. Please check with your official contact." In this way, the user can quickly respond to the risk of fraud.

[0245] This system will enable users to receive real-time warnings about fraud and criminal activity, which is expected to help prevent people from falling victim to fraud.

[0246] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0247] Step 1:

[0248] A user launches an app on their smartphone and starts the voice capture function. When the user starts a phone call or face-to-face conversation, the device starts capturing voice data in real time. The input is the user's voice data, and the output is the captured voice data.

[0249] Step 2:

[0250] The captured voice data is sent to the Google Cloud Speech-to-Text API, where it is converted into text data using speech recognition technology. The input is voice data, and the output is converted text data. This conversion process analyzes the linguistic characteristics of the voice data and the speaker's voice and converts it into text.

[0251] Step 3:

[0252] The converted text data is sent to a server and analyzed by a generative AI model (OpenAI GPT). The input is text data and the output is the analysis result. In this step, the text data is matched with fraud patterns to evaluate the likelihood of fraud or criminal activity. Specifically, the model is queried using a specific prompt sentence.

[0253] Rate the following text as a scam: Your credit card details may have been stolen. Check now

[0254] Step 4:

[0255] The server generates a warning message and countermeasure suggestions based on the analysis results. The input is the analysis results, and the output is a warning message and countermeasure suggestions. If the analysis results indicate a possibility of fraud, a warning message is created and saved as data to be sent to the user.

[0256] Step 5:

[0257] The generated warning message and suggested measures are sent to the user's smartphone using the Twilio API. The input is the warning message and suggested measures, and the output is the warning message displayed on the user's smartphone. Specifically, an SMS is sent via the Twilio API, and a pop-up message is displayed on the user's device.

[0258] Step 6:

[0259] After the conversation is over, all collected data is sent to the server and stored in a database. The input is the conversation data and analysis results, and the output is the data stored in the database. The stored data is used to retrain the generative AI model, contributing to improving the accuracy of the system.

[0260] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0261] This invention is a system that converts voice data into text data in real time, evaluates the possibility of fraud or criminal activity using a generative artificial intelligence model and an emotion engine, and presents a warning and countermeasure suggestions to the user. Specific embodiments are described below.

[0262] System configuration and operation

[0263] 1. The user launches the app

[0264] Users launch a dedicated app on their smartphone or tablet.

[0265] The device will display the app interface and begin voice capture and emotion recognition functions.

[0266] 2. Real-time conversation transcription and emotion recognition

[0267] The user answers a phone call or starts a face-to-face conversation. The device begins capturing audio through the microphone.

[0268] The device converts voice data into text data in real time, while the emotion engine simultaneously recognizes emotions from the user's voice and facial expressions.

[0269] 3. Sending text data and emotion data

[0270] The device then transmits the generated text data and emotion data to a server, where the data is encrypted using a secure communication protocol.

[0271] 4. Analysis of text data and emotion data

[0272] The server inputs the received text data into a generative AI model, while inputting the emotion data into an emotion evaluation engine.

[0273] The generative AI model analyzes the conversation content against a database of fraud and criminal patterns, while an emotion assessment engine simultaneously analyzes the user's emotional state.

[0274] 5. Generating comprehensive evaluation results

[0275] The server integrates the analysis results of the generative AI model and the emotion evaluation engine to comprehensively assess the likelihood of fraud or criminal activity.

[0276] If the emotional data indicates strong negative emotions such as anxiety or fear in the user, the importance of the warning and the priority of the countermeasures will be adjusted.

[0277] 6. Providing warnings and suggested solutions

[0278] The server generates a warning message and a countermeasure proposal and sends them to the terminal.

[0279] The device will display a warning to the user as a pop-up message and suggest appropriate measures, such as "This may be a scam" or "Please check with official contacts."

[0280] 7. Recording speech patterns and emotional data

[0281] When the conversation is finished, the terminal sends all data to the server.

[0282] The server records the received data and stores it in a back-end database, which can be used to retrain future AI models and emotion engines.

[0283] 8. Retraining AI models and emotion engines

[0284] The server periodically retrains the generative AI model and emotion engine with new speech patterns and emotion data, thereby improving the accuracy of the model and engine.

[0285] Specific examples

[0286] Scenario: Fake billing call

[0287] 1. Launch the app

[0288] An elderly user launches a dedicated app.

[0289] The user prepares to receive the call.

[0290] 2. Conversation initiation, transcription, and emotion recognition

[0291] The user receives a call and the conversation begins with the user being told that the outstanding amount is due and that they should pay it immediately.

[0292] The device captures conversations in real time and instantly transcribes them into text, while simultaneously detecting anxiety from the user's tone and content of voice.

[0293] 3. Data submission and analysis

[0294] The terminal transmits the generated text data and emotion data to the server.

[0295] The data received by the server is analyzed using a generative AI model and emotion evaluation engine.

[0296] A generative AI model detects fraud patterns such as "non-payment" and "pay now," while a sentiment assessment engine detects high levels of anxiety in users.

[0297] 4. Comprehensive evaluation and warning message generation

[0298] The server determines that there is a high possibility of fraud and generates a warning message and a suggestion to "check with an official contact." Based on the emotional data, the importance of the warning is increased, urging even more caution.

[0299] 5. Displaying warnings and suggested solutions

[0300] The terminal displays these messages to the user as popups.

[0301] 6. Record data and retrain

[0302] After the conversation ends, the device sends all data to the server.

[0303] The server records the data and uses it to retrain the generative AI model and emotion engine.

[0304] This system not only effectively protects users from fraud and criminal activity, but also suggests countermeasures based on the user's emotional state. This increases users' sense of security and helps prevent fraud from occurring. Furthermore, the accuracy of the system can be constantly improved based on the collected data.

[0305] ---

[0306] The above is an explanation of the "Mode for Carrying Out the Invention" based on the scope of the patent claims for the invention that combines an emotion engine. This content will help you understand the specific implementation methods of the invention and you can use it as a reference to carry out the invention.

[0307] The processing flow will be explained below.

[0308] Step 1:

[0309] The user launches the dedicated app on their smartphone or tablet, which displays the app's interface and starts the voice capture and emotion recognition functions.

[0310] Step 2:

[0311] When a user answers a phone call or starts a face-to-face conversation, the device begins capturing audio through the microphone. Audio capture begins immediately and the captured audio data is converted into text data in real time. At the same time, the emotion engine recognizes emotions from the user's voice and facial expressions in real time.

[0312] Step 3:

[0313] The device then transmits the generated text data and emotion data to a server, where the data is encrypted using a secure communication protocol.

[0314] Step 4:

[0315] The server inputs the received text data into a generative AI model and begins analysis. Meanwhile, it inputs the emotion data into an emotion evaluation engine to evaluate the user's emotional state.

[0316] Step 5:

[0317] The generative AI model analyzes the text data by comparing it with a database of fraud and criminal patterns, while the emotion assessment engine evaluates the user's emotional state, particularly anxiety and fear, and integrates this data into the generative AI model's analysis.

[0318] Step 6:

[0319] The server combines the analysis results of the generated AI model and the emotion evaluation engine to comprehensively evaluate the likelihood of fraud or criminal activity, adjust the importance and content of warning messages based on the emotion data, and generate the final evaluation result.

[0320] Step 7:

[0321] The server generates a warning message and suggested countermeasures and sends them to the device. The device receives the message and displays a pop-up warning to the user, suggesting appropriate countermeasures, such as "This may be a scam" or "Please check with the official contacts."

[0322] Step 8:

[0323] When the user ends the call, the device transmits all conversation data (voice and transcription data) and emotion data to the server, where the data is again encrypted and transmitted via a secure communication protocol.

[0324] Step 9:

[0325] The server records the received data and stores it in a back-end database, which then uses it to retrain the generative AI model and emotion engine.

[0326] Step 10:

[0327] The server periodically retrains the generative AI model and emotion engine based on new speech patterns and emotion data, thereby improving the accuracy of the model and engine.

[0328] ---

[0329] The above is a concrete explanation of the processing steps of the invention that combines the emotion engine. This allows you to clearly understand how the system operates and the specific processing that is performed.

[0330] Example 2

[0331] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0332] Conventional systems could convert voice data into text data in real time to assess the likelihood of fraud or criminal activity, but they were unable to adjust the severity of warnings or the priority of countermeasures while taking the user's emotional state into account. As a result, it was not possible to propose appropriate countermeasures based on the user's emotional state, making it difficult to sufficiently reduce the risk of fraud or criminal activity. The purpose of this invention is to provide an effective fraud and criminal activity assessment system that also takes the user's emotional state into account.

[0333] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing voice data and text data and recognizing the user's emotional state in real time, a means for analyzing the text data using a generative artificial intelligence model and evaluating the possibility of fraud or criminal activity, and a means for integrating the analysis results and the user's emotional state to adjust the importance of the warning and the priority of countermeasures. This makes it possible to more accurately evaluate the possibility of fraud or criminal activity and to provide appropriate warnings and propose countermeasures according to the user's emotional state.

[0334] "Voice data" refers to digital audio information captured from a user's speech via a microphone.

[0335] "Text data" is digital document information that is created by converting voice data into text in real time.

[0336] A "generative artificial intelligence model" is a machine learning algorithm used to analyze text data and understand specific patterns and meanings.

[0337] "Emotional state" refers to the psychological and emotional state of the user, which is analyzed from the user's tone of voice, facial expression, etc.

[0338] "Real-time" refers to operations and processing occurring instantly with minimal delay.

[0339] "Fraud and criminal activity" refers to any activity that involves using dishonest means to deceive others for profit or that violates the law.

[0340] "Evaluation means" refers to techniques or methods for analyzing specific information based on input data and deriving results.

[0341] A "warning message" is a notification message that alerts the user and notifies them of the existence of a risk.

[0342] A "measure proposal" is a proposal that instructs the user on specific actions and methods to take in response to a detected risk.

[0343] "Encryption means" is a technology that encrypts data using a certain algorithm to transmit the data securely, making the contents of the data unintelligible to third parties.

[0344] A "server" is a computer system for storing, analyzing, and providing data.

[0345] A "terminal" is a device (e.g., a smartphone or tablet) that a user directly operates and that communicates with a server.

[0346] A "secure communication protocol" is a standardized procedure or rule for securely communicating data (e.g., HTTPS).

[0347] This invention is a system that converts voice data into text data in real time, uses a generative artificial intelligence model and an emotion evaluation engine to evaluate the possibility of fraud or criminal activity, and presents warnings and suggested countermeasures to the user.

[0348] System configuration and operation

[0349] 1. Launch the app

[0350] The user launches the dedicated app on their smartphone or tablet. The device displays the app's interface and is ready to begin voice capture and emotion recognition. The app has a user-friendly interface, including a button that says "Start Call."

[0351] 2. Audio data capture and conversion

[0352] When a user answers a call, the device's microphone captures the audio data, which is then converted into text data in real time using the Google Cloud Speech-to-Text API, instantly recording what the user is saying.

[0353] 3. Emotion recognition

[0354] At the same time, the system uses the Microsoft Azure Cognitive Services Emotion API to analyze the user's emotions in real time from the tone and content of their voice. This API recognizes the speaker's emotions from voice data and can identify negative emotions such as anxiety or fear.

[0355] 4. Data transmission

[0356] The device encrypts the generated text data and emotion data using a secure communication protocol (e.g., HTTPS) and transmits it to the server, where it is protected from unauthorized access.

[0357] 5. Data Analysis

[0358] The server inputs the received text data into OpenAI's GPT-4 model to evaluate the likelihood of fraud or criminal activity, and analyzes the emotional data using Azure's Emotion API. This allows for highly accurate detection of fraud and criminal activity based on the user's emotional state and text data.

[0359] 6. Generating comprehensive evaluation results

[0360] The server integrates the analysis results of the generative AI model and the emotion evaluation engine to comprehensively evaluate the likelihood of fraud or criminal activity, and adjusts the importance of warnings and the priority of countermeasures based on the level of anxiety or fear expressed by the user as indicated by the emotion data.

[0361] 7. Providing warnings and suggested solutions

[0362] The server generates a warning message and suggested actions and sends it to the device, which then displays these messages to the user as a pop-up notification, such as "This may be a scam" or "Please check with an official contact."

[0363] 8. Record data and retrain

[0364] After the conversation is over, the device sends all data to the server, which stores it in a back-end database. This stored data will be used to retrain the generative AI model and emotion engine in the future. Periodic retraining ensures the system stays up to date with the latest fraud patterns and the user's emotional state.

[0365] Specific examples

[0366] Scenario: Fake billing call

[0367] Launching the app

[0368] The elderly user launches the dedicated app and prepares to receive a call. The device then starts the voice capture function.

[0369] Conversation initiation, transcription, and emotion recognition

[0370] The user receives a call and the conversation begins with the message, "Your payment is outstanding and you need to pay it immediately." The device captures the conversation in real time and instantly transcribes it into text using the Google Cloud Speech-to-Text API. At the same time, it uses Azure's Emotion API to detect anxiety from the user's tone and content of voice.

[0371] Data transmission and analysis

[0372] The device sends the generated text data and emotion data to a server, which then analyzes the received data using OpenAI's GPT-4 model and Azure Emotion API. The GPT-4 model detects fraudulent patterns such as "unpaid" and "pay immediately," while the emotion evaluation engine detects high levels of anxiety in the user.

[0373] Evaluation and warning message generation

[0374] The server determines that there is a high possibility of fraud and generates a warning message and a suggestion to "check with an official contact." Based on the emotional data, the importance of the warning is increased, urging even more caution.

[0375] Displaying warnings and suggested solutions

[0376] The terminal displays these messages to the user as pop-up notifications.

[0377] Recording Data and Retraining

[0378] After the conversation ends, the device sends all data to the server, which records it and uses it to retrain the generative AI model and emotion engine.

[0379] Prompt Sentence Examples

[0380] What should you do if you receive a call saying, "Please transfer the money right now"?

[0381] By inputting this prompt into a generative AI model, the system can suggest appropriate countermeasures.

[0382] The above is an embodiment of the present invention. This system effectively protects users from fraud and criminal activity and provides countermeasure suggestions according to the user's emotional state.

[0383] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0384] Step 1:

[0385] A user launches the dedicated app on a smartphone or tablet. The device displays the app's interface and prepares to start the voice capture and emotion recognition functions. The input is the user's operation, and the output is the display of the app's home screen, which includes a "Start phone call" button.

[0386] Step 2:

[0387] When a user receives a call, the device's microphone captures audio data. The input is the captured audio data, and the output is a real-time stream of that audio data. The device then sends the captured audio data to the Google Cloud Speech-to-Text API, which converts it into text data. Specifically, the audio data is analyzed in real time and instantly converted into text.

[0388] Step 3:

[0389] At the same time, the device uses the Emotion API from Microsoft Azure Cognitive Services to analyze the user's emotions in real time. The input is the captured voice data, and the output is the emotion recognition results. Specifically, the Emotion API identifies negative emotions from the tone and content of the voice.

[0390] Step 4:

[0391] The text data and emotion data generated by the device are encrypted using a secure communication protocol (e.g., HTTPS) and sent to the server. The input is text data and emotion data, and the output is the encrypted data. This ensures data security.

[0392] Step 5:

[0393] The server inputs the received text data into OpenAI's GPT-4 model to evaluate the likelihood of fraud or criminal activity. The input is the received text data, and the output is the analysis result. Specifically, the GPT-4 model detects keywords such as "unpaid" and "pay immediately" in the text.

[0394] Step 6:

[0395] The server analyzes the emotion data using an emotion evaluation engine. The input is the received emotion data, and the output is the analyzed emotion evaluation result. Specifically, the Emotion API evaluates the user's anxiety or fear level.

[0396] Step 7:

[0397] The server integrates the analysis results of the generative AI model and the emotion evaluation engine to generate a comprehensive evaluation result. The input is the analysis results of the text data and the evaluation results of the emotion data, and the output is a comprehensive evaluation report. Specifically, the server combines both sets of data to evaluate the likelihood and severity of fraud or criminal activity.

[0398] Step 8:

[0399] The server generates a warning message and suggested countermeasures based on the evaluation results and sends them to the terminal. The input is the overall evaluation result, and the output is a warning message and suggested countermeasures. Specifically, the server generates messages such as "This may be a scam" or "Please check with the official contact point."

[0400] Step 9:

[0401] The terminal displays the received warning message and suggested countermeasures to the user as a pop-up notification. The input is the warning message and suggested countermeasures sent from the server, and the output is the notification to the user. Specifically, the terminal displays a pop-up message on the screen to immediately notify the user of the warning.

[0402] Step 10:

[0403] After the conversation is over, the device sends all data to the server. The input is the generated text data and emotion data, and the output is data sent to the server. The server stores this data in a backend database. The stored data is used to retrain the generative AI model and emotion engine. This improves the accuracy of the system and allows it to adapt to the latest fraud patterns and the user's emotional state.

[0404] (Application example 2)

[0405] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0406] Fraud and criminal activity methods are evolving daily, resulting in a constant stream of fraud victims perpetrated over the phone or in face-to-face conversations. Elderly people and those with low digital literacy are particularly susceptible to scams, and are prone to financial loss and psychological damage. However, conventional fraud prevention systems struggle to provide real-time countermeasures and are unable to provide warnings or suggest countermeasures that take into account the user's emotional state. To solve this problem, a system is needed that converts speech into text in real time, uses a generative artificial intelligence model and an emotion engine to comprehensively evaluate the likelihood of fraud or criminal activity, and then provides appropriate warnings and suggests countermeasures to the user.

[0407] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting voice data into text data in real time, means for analyzing the text data using a generative artificial intelligence model and evaluating the possibility of fraud or criminal activity, and means for recognizing the user's emotional state using an emotion engine and displaying a warning message and suggested countermeasures to the user based on the evaluation results. This makes it possible to detect fraud or criminal activity in real time and issue appropriate warnings and suggest countermeasures according to the user's emotional state.

[0408] "Audio data" refers to audio signals uttered by a user recorded as digital information.

[0409] "Text data" is a collection of character information generated by analyzing voice data.

[0410] A "generative artificial intelligence model" is an algorithm that uses machine learning to analyze text data and understand and evaluate specific patterns and meanings.

[0411] The "emotion engine" is an algorithm that analyzes the user's emotional state from their voice and facial expressions and evaluates the results.

[0412] A "warning message" is an informational notice intended to inform the user of a high possibility of fraud or criminal activity.

[0413] "Measures Proposal" is an information notification that indicates specific actions or measures that the user should take based on the warning message.

[0414] A "server" is a computer system that analyzes voice data and text data and generates comprehensive evaluation results.

[0415] A "terminal" is a device used by a user, which has the function of capturing voice data and displaying warning messages and suggested countermeasures received from the server.

[0416] This invention is a system that converts voice data into text data in real time, evaluates the possibility of fraud or criminal activity using a generative AI model and an emotion engine, and presents warnings and countermeasure suggestions to users. This system is mainly composed of a server, a terminal, and a user.

[0417] System configuration

[0418] Terminal

[0419] The terminal is a device such as a smartphone or tablet that has a microphone for capturing audio data and a display for displaying text data and warning messages.

[0420] When a user initiates a phone or face-to-face conversation, the device captures voice data and converts it into text data in real time using a speech recognition engine.

[0421] server

[0422] The server is a central computer system that receives and analyzes the text data and emotion data sent from the terminals.

[0423] To analyze the text data, a generative AI model (such as BERT or GPT) is used. The generative AI model compares patterns of fraud and criminal activity with a database to assess whether the conversation is fraudulent. In addition, an emotion engine analyzes the user's emotional state based on the voice data sent from the device. This makes it possible to determine whether the user is feeling anxious or fearful.

[0424] User

[0425] A user is an individual who uses the system and is the sender of voice data. When a user answers a phone call or has a face-to-face conversation, the voice data is captured through the terminal and sent to the server. Analysis results and warning messages from the server are notified to the user through the terminal.

[0426] Program processing

[0427] 1. Capture and convert voice data: Capture voice using the device's microphone and convert it into text data in real time using the Google Speech Recognition API.

[0428] 2. Data Analysis: Send the text data and emotion data to the server, where the text data is analyzed using a generative AI model (e.g., BERT), and the emotion engine is used to analyze the user's emotional state.

[0429] 3. Evaluation and Notification: The server integrates the results of analysis by the generative AI model and emotion engine, and if there is a high possibility of fraud, a warning message is generated and sent to the device. The notification content may be something like "This is likely a scam" or "Please check with the official contact point."

[0430] Specific examples

[0431] Scenario: Fake billing call

[0432] 1. User receives a call: The elderly user receives a call and launches the app on their device.

[0433] 2. Real-time transcription of the conversation: The audio data of "It's unpaid and please pay it immediately" is captured and converted into text data in real time.

[0434] 3. Data transmission and analysis: Text data and emotion data are transmitted to the server, and the generative AI model detects fraud patterns such as "unpaid" and "pay immediately." At the same time, the emotion engine detects high levels of anxiety from the user.

[0435] 4. Integration of evaluation results and generation of warning: The server determines that there is a high possibility of fraud and generates a warning message such as "Please check with the official contact point."

[0436] 5. Warning Notifications: The device will display these messages to the user as a popup.

[0437] Prompt Sentence Examples

[0438] "Pay any outstanding amounts immediately."

[0439] Category labels: Scam, Safety

[0440] In this way, the system can provide users with real-time warnings and countermeasures against fraud and criminal activity, helping to keep them safe.

[0441] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0442] Step 1:

[0443] A user launches an app on a smartphone or tablet and starts a phone or face-to-face conversation. The device's microphone captures audio data. The input is the user's voice data, which is recorded as a digital audio signal by the app's microphone. The output is the captured voice data.

[0444] Step 2:

[0445] The device converts captured voice data into text data in real time. This conversion is performed using a speech recognition engine (e.g., Google Speech Recognition API). The input is the captured voice data, and the output is the converted text data.

[0446] Step 3:

[0447] The converted text data and emotion data detected from the user's voice tone and content are sent from the device to the server. The data is encrypted and sent using a secure communication protocol (e.g., SSL / TLS). The input is text data and emotion data, and the output is data that is sent securely.

[0448] Step 4:

[0449] The server analyzes the received text data using a generative AI model (e.g., BERT). This analysis evaluates the text data to detect patterns of fraud and criminal activity. The input is the submitted text data, and the output is the generative AI model's evaluation of the fraud and criminal activity.

[0450] Step 5:

[0451] In parallel, the server inputs the received emotion data into the emotion engine to analyze the user's emotional state. The emotion engine evaluates the user's negative emotions, such as anxiety, discomfort, and fear, from the voice data and text. The input is the transmitted emotion data, and the output is the analysis result of the user's emotional state.

[0452] Step 6:

[0453] The server integrates the evaluation results of the generative AI model and the analysis results of the emotion engine to comprehensively determine the likelihood of fraud or criminal activity. Depending on the evaluation results, a warning message and countermeasure proposals are generated. The input is the analysis results of text data and emotion data, and the output is the integrated evaluation results and the warning message and countermeasure proposals based on them.

[0454] Step 7:

[0455] The server generates a warning message and sends it to the terminal. The input is the warning message and the countermeasure proposal, and the output is the notification data that is securely transmitted.

[0456] Step 8:

[0457] The terminal displays a warning message and suggested countermeasures to the user as a pop-up message. This allows the user to receive warnings about possible fraud or criminal activity and appropriate countermeasures in real time. The input is the warning message and suggested countermeasures sent from the server, and the output is the pop-up message displayed on the terminal display.

[0458] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0459] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0460] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0461] [Second embodiment]

[0462] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0463] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0464] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0465] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0466] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0467] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0468] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0469] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0470] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0471] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0472] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0473] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0474] This invention is a system that converts voice data into text data in real time, evaluates the possibility of fraud or criminal activity using a generative artificial intelligence model, and presents a warning and countermeasure suggestions to the user. Specific embodiments will be described below.

[0475] System configuration and operation

[0476] 1. The user launches the app

[0477] Users launch a dedicated app on their smartphone or tablet.

[0478] The device will display the app interface and start the audio capture function.

[0479] When a user starts a phone or in-person conversation, the app automatically begins capturing audio.

[0480] 2. Real-time transcription of conversations

[0481] The device converts the captured audio into text data in real time, using voice recognition technology to instantly convert spoken words into written text.

[0482] Once the transcription is complete, the text data is sent to the server.

[0483] 3. Analysis and Evaluation of Text Data

[0484] The server inputs the received text data into a generative artificial intelligence model.

[0485] The generative AI model analyzes the content of the conversation by comparing it with a database of patterns of fraud and criminal activity.

[0486] If fraud or criminal activity is suspected, the server evaluates the results and generates a warning message.

[0487] 4. Present warnings and suggested solutions

[0488] The server returns data including a warning message and suggested countermeasures to the terminal.

[0489] The device will warn the user with a pop-up message and suggest appropriate measures, such as "This may be a scam" or "Please check with official contacts."

[0490] 5. Recording conversation patterns

[0491] The terminal sends the data of the completed conversation to the server, where it is recorded in the backend.

[0492] The server analyzes the stored data and uses it as data for retraining the generative AI model.

[0493] Specific examples

[0494] Scenario: Fake billing call

[0495] 1. Launch the app

[0496] An elderly user launches a dedicated app.

[0497] The user prepares to receive the call.

[0498] 2. Start a conversation and transcribe

[0499] The user receives a call and the conversation begins with the user being told that the outstanding amount is due and that they should pay it immediately.

[0500] The device captures the conversation in real time and instantly transcribes it into text.

[0501] 3. Sending and analyzing text data

[0502] The terminal transmits the generated text data to the server.

[0503] The data received by the server is analyzed using a generative AI model to detect fraudulent patterns such as "unpaid" and "pay immediately."

[0504] 4. Displaying warnings and suggested solutions

[0505] The server determines that there is a high possibility of fraud and generates a warning message and a suggestion to "check with the official contact point."

[0506] The terminal displays these messages to the user as popups.

[0507] 5. Recording conversation patterns

[0508] After the conversation is over, the device sends all data to the server.

[0509] The server records the data and uses it to retrain the generative AI model.

[0510] This system not only effectively protects users from fraud and criminal activity, but also allows the accuracy of the generative AI model to be constantly improved based on the collected data.The system covers both online and offline risks, making it effective for a wide range of users.

[0511] ---

[0512] The above is the explanation of the "Mode for Carrying Out the Invention" based on the scope of the claims. This content will help you understand the specific methods for carrying out the invention and you can use it as a reference to carry out the invention.

[0513] The processing flow will be explained below.

[0514] Step 1:

[0515] The user launches the dedicated app on their smartphone or tablet. The device displays the app interface and starts the audio capture function.

[0516] Step 2:

[0517] The user answers a phone call or starts a face-to-face conversation. The device begins capturing audio through the microphone. Once audio capture begins, the device transcribes the captured audio data in real time.

[0518] Step 3:

[0519] The device generates the transcribed text data and sends it to the server, where it is encrypted using a secure communication protocol.

[0520] Step 4:

[0521] The server inputs the received text data into a generative AI model, which then compares it with a database of fraud and criminal patterns and analyzes the conversation.

[0522] Step 5:

[0523] The generative AI model returns the analysis results to the server, which evaluates them and determines whether there is any potential for fraud or criminal activity.

[0524] Step 6:

[0525] The server generates a warning message based on the evaluation results, including specific warning content and suggested countermeasures for the user.

[0526] Step 7:

[0527] The server sends back data containing a warning message and suggested solutions to the terminal, which receives it and displays it to the user as a pop-up message.

[0528] Step 8:

[0529] The user ends the call. The device sends all conversation data (audio and transcription data) to the server.

[0530] Step 9:

[0531] The server records the data it receives and stores it in a back-end database, which can then be used to retrain the AI ​​model in the future.

[0532] Step 10:

[0533] The server periodically retrains the generative AI model based on new conversation patterns, thereby improving the model's detection accuracy.

[0534] ---

[0535] This is a detailed explanation of the program's processing steps, which will give you a clear understanding of how the system works and what specific processes are performed.

[0536] Example 1

[0537] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0538] Conventional methods have had difficulty detecting fraud and criminal activity in real time and quickly warning users. They also lacked the means to securely transmit voice data and properly return analysis results and warning messages to the device. Therefore, there was a need for an effective means for users to prevent themselves from becoming victims of fraud and criminal activity.

[0539] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0540] In this invention, the server includes means for converting voice data into text data in real time, means for analyzing the text data using a generative artificial intelligence model to evaluate the likelihood of fraud or criminal activity, means for displaying a warning message and suggested countermeasures to the user based on the evaluation results, means for transmitting text data transmitted from the user to the terminal to the server, means for analyzing the text data using the generative artificial intelligence model to detect the likelihood of fraud or criminal activity, means for returning the generated warning message and suggested countermeasures to the terminal, and means for recording the conversation data and using it as data for retraining, thereby enabling users to be effectively protected from fraud and criminal activity.

[0541] "Audio data" means data in digital form that contains human speech or other sound information.

[0542] "Text data" is data consisting of characters and symbols that is obtained as a result of converting voice data.

[0543] A "generative artificial intelligence model" is an artificial intelligence model with natural language processing capabilities that is trained based on a large dataset.

[0544] "Analysis" is the process performed to determine the possibility of fraud or criminal activity based on the input data.

[0545] A "warning message" is a message that is generated as a result of analysis to warn or alert the user.

[0546] "Measure suggestions" are specific actions or measures recommended to the user based on the warning message.

[0547] "User" refers to any individual or corporation that uses this system.

[0548] "Terminal" refers to an electronic device, such as a smartphone or tablet, that allows a user to access and use the system.

[0549] A "server" is a central computer system used to process voice and text data and provide information to users.

[0550] "Real-time" refers to processing that occurs almost simultaneously with the occurrence of an event.

[0551] "Conversation data" refers to data that includes captured speech and text data generated based on the speech.

[0552] "Retraining" is the process of retraining a generative artificial intelligence model using new data to improve its accuracy.

[0553] MODE FOR CARRYING OUT THE INVENTION

[0554] The present invention provides a system that converts voice data into text data in real time, evaluates the possibility of fraud or criminal activity using a generative artificial intelligence model, and presents a warning and countermeasure suggestions to the user. A specific embodiment of the present invention will be described below.

[0555] System configuration and operation

[0556] The user launches the app

[0557] Users launch a dedicated app on their smartphone or tablet. This app has a voice capture function, and when the user starts a phone or face-to-face conversation, the device begins capturing the voice in real time.

[0558] Voice capture to text conversion

[0559] The device converts the captured voice data into text data in real time using voice recognition software (e.g., Google Speech-to-Text API), and the converted text data is immediately sent to the server.

[0560] Text data analysis

[0561] The server inputs the received text data into a generative AI model (e.g., OpenAI GPT-4). The generative AI model analyzes the conversation by comparing it with a database of fraudulent and criminal patterns. If the analysis reveals a possibility of fraud or criminal activity, the server generates a warning message and suggests countermeasures.

[0562] Warnings and suggested solutions

[0563] The server generates a warning message and sends it back to the device, which displays it to the user as a pop-up message. Specific messages include "This may be a scam" and "Please check with your official contacts."

[0564] Recording and retraining conversation data

[0565] After the conversation is complete, the device sends all conversation data to the server, which records it in a database. This data is used to retrain the generative AI model, helping to improve the accuracy of the system.

[0566] Specific operation example

[0567] Scenario: Fake billing call

[0568] 1. Launch the app

[0569] An elderly user launches a dedicated app.

[0570] The user prepares to receive the call.

[0571] 2. Start a conversation and transcribe

[0572] The user receives a call and the conversation begins with the user being told that the outstanding amount is due and that they should pay it immediately.

[0573] The device captures the conversation in real time and instantly transcribes it into text.

[0574] 3. Sending and analyzing text data

[0575] The terminal transmits the generated text data to the server.

[0576] The data received by the server is analyzed using a generative artificial intelligence model to detect fraudulent patterns such as "non-payment" and "immediate payment."

[0577] 4. Displaying warnings and suggested solutions

[0578] The server determines that there is a high possibility of fraud and generates a warning message and a suggestion to "check with the official contact point."

[0579] The terminal displays these messages to the user as popups.

[0580] 5. Recording conversation patterns

[0581] After the conversation is over, the terminal sends all data to the server.

[0582] The server records the data and uses it to retrain the generative AI model.

[0583] In this way, the system of the present invention can consistently convert voice data in real time, analyze it using a generative AI model, and promptly warn users and suggest countermeasures. This system effectively protects users from fraud and criminal activity, and the accuracy of the system is continuously improved.

[0584] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0585] Step 1:

[0586] The user launches the app

[0587] Input: The user operates a smartphone or tablet device to launch a dedicated app.

[0588] What it does: The device displays the app interface and puts the audio capture feature into standby.

[0589] Output: The audio capture feature is enabled and audio is recorded when the user starts speaking.

[0590] Step 2:

[0591] The device captures audio in real time

[0592] Input: Ambient sounds, especially phone calls and face-to-face conversations you have.

[0593] What it does: Your device uses the app's audio capture feature to record audio data in real time.

[0594] Output: Recorded audio data.

[0595] Step 3:

[0596] The device converts the voice data into text data.

[0597] Input: Real-time recorded audio data.

[0598] How it works: Your device uses speech recognition software (e.g., Google Speech-to-Text API) to convert voice data into text.

[0599] Output: The converted text data.

[0600] Step 4:

[0601] The device sends text data to the server.

[0602] Input: The converted text data.

[0603] How it works: The device sends text data to the server using a secure communication protocol (e.g., HTTPS).

[0604] Output: Text data is sent to the server.

[0605] Step 5:

[0606] The server analyzes the text data using a generative AI model

[0607] Input: The text data sent to the server.

[0608] How it works: The server feeds text data into a generative artificial intelligence model (e.g., OpenAI GPT-4) to analyze it for potential fraud or criminal activity.

[0609] Output: Analysis results in assessment data of the likelihood of fraud or criminal activity.

[0610] Step 6:

[0611] The server evaluates the results and generates a warning message

[0612] Input: Analysis results generated by a generative AI model.

[0613] How it works: The server evaluates the analysis results and generates a warning message and suggested actions if there is a high possibility of fraud or criminal activity.

[0614] Output: Generated warning messages and suggested actions.

[0615] Step 7:

[0616] The server sends a warning message and suggested solutions back to the device.

[0617] Input: Generated warning message and suggested action.

[0618] Operation: The server returns a warning message and suggested solutions to the device as an HTTP response.

[0619] Output: A warning message and suggested solutions will be sent to your device.

[0620] Step 8:

[0621] The device displays a warning and suggests countermeasures to the user.

[0622] Input: The warning message and suggested action returned by the server.

[0623] How it works: The device displays a pop-up notification to the user with a warning message and suggested solutions.

[0624] Output: User confirms warning message and suggested action.

[0625] Step 9:

[0626] The device records conversation patterns and sends them to the server.

[0627] Input: Finished conversation data.

[0628] Operation: The terminal records the conversation data that has been completed and sends it to the server.

[0629] Output: Conversation data recorded on the server.

[0630] Step 10:

[0631] The server records the data and uses it to retrain the generative AI model.

[0632] Input: Conversation data sent to the server.

[0633] How it works: The server stores conversation data and uses it as retraining data for generative AI models.

[0634] Output: An updated generative AI model and conversation data recorded in a database.

[0635] (Application example 1)

[0636] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0637] In modern society, fraud and criminal acts are becoming increasingly sophisticated, and many people are falling victim to these methods. To address this situation, a system is needed that can detect fraud and criminal acts early and provide appropriate warnings and countermeasures. However, current technology does not provide a system that can analyze voice data in real time and immediately evaluate the possibility of fraud and provide a warning. Therefore, an objective of the present invention is to provide a system that can detect fraud and criminal acts in real time and provide a warning, allowing users to communicate safely.

[0638] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0639] In this invention, the server includes means for converting voice data into text data in real time, means for analyzing the text data using a generative artificial intelligence model to evaluate the possibility of fraud or criminal activity, means for displaying a warning message and countermeasure suggestions to the user based on the evaluation results, means for providing a voice capture function, means for converting voice into text using voice recognition technology, means for analyzing the text by matching it with fraud patterns, and means for sending a warning message to the user using communication means, thereby enabling the user to be aware of the possibility of fraud or criminal activity in real time and take appropriate countermeasures.

[0640] "Audio data" refers to continuous sound signals obtained from conversations, communications, etc.

[0641] "Text data" is voice data converted into a string of characters in a language.

[0642] A "generative artificial intelligence model" is an AI technology that learns large amounts of data and generates and analyzes sentences and information.

[0643] "Voice recognition technology" is a technology that converts voice data into text in real time.

[0644] "Fraud patterns" are data that indicate combinations of words and phrases associated with fraud.

[0645] A "warning message" is a notification that alerts the user to danger or problems.

[0646] "Measure suggestions" are advice that shows specific actions or methods that users should take.

[0647] "User" refers to any person or organization that uses this system.

[0648] The "audio capture function" is a function for collecting audio in real time.

[0649] "Communication means" refers to the technology and protocols used to exchange information.

[0650] A "server" is a computer system that processes and stores data and provides services over a network.

[0651] "Terminal" refers to a device or equipment that is directly operated by a user.

[0652] "Analysis results" refers to the results of analysis using a generative artificial intelligence model.

[0653] MODE FOR CARRYING OUT THE INVENTION

[0654] This invention is a security system for users that converts voice data into text data in real time, analyzes the text data using a generative artificial intelligence model to evaluate the possibility of fraud or criminal activity, and provides the user with a warning and suggestions for countermeasures. This system is mainly composed of the following elements.

[0655] 1. Hardware and Software Selection

[0656] The following hardware and software are used to realize this system.

[0657] Smartphone terminal: A device operated by a user.

[0658] Microphone: Used to capture audio data.

[0659] Server: Processes data and manages the database.

[0660] Google Cloud Speech-to-Text API: Provides speech recognition technology.

[0661] Generative AI models (e.g., OpenAI GPT): Analyze text data to assess the likelihood of fraud or criminal activity.

[0662] Twilio API: A communication method for sending alert messages to users.

[0663] 2. Explanation of program processing

[0664] The main processing flow of the program of this system is explained below.

[0665] User voice input

[0666] The user starts a conversation using a smartphone. The device's voice capture function is activated, capturing the conversation's audio data in real time. This allows the user's conversation to be input into the system in real time and prepared for analysis.

[0667] Converting audio data to text

[0668] The server converts the captured audio data into text data using the Google Cloud Speech-to-Text API, which then sends the audio data to the server as text data.

[0669] Text data analysis

[0670] The server uses a generative AI model (OpenAI GPT) to analyze the converted text data. This identifies patterns of fraud and criminal activity and assesses risk. This analysis process is performed by using specific keywords and phrases as prompts. The following are example prompts:

[0671] Rate the following text as a scam: Your credit card details may have been stolen. Check now

[0672] Warning messages and suggested solutions

[0673] If the server determines based on the analysis results that there is a high possibility of fraud or criminal activity, it uses the Twilio API to send a warning message to the user's device. The warning message also includes specific suggestions for countermeasures. For example, a message such as "This may be a scam. Please contact an official contact point" may be displayed.

[0674] Adding specific examples

[0675] As a concrete example, let's consider a scenario in which an elderly user receives a fraudulent phone call. When the user receives a call stating, "You have an outstanding payment and need to pay it immediately," the audio is captured in real time and converted to text using the Google Cloud Speech-to-Text API. The OpenAI GPT model then evaluates the likelihood of fraud and, if it determines there is a high risk, sends a warning message to the user's smartphone via the Twilio API stating, "There is a high possibility of fraud. Please check with your official contact." In this way, the user can quickly respond to the risk of fraud.

[0676] This system will enable users to receive real-time warnings about fraud and criminal activity, which is expected to help prevent people from falling victim to fraud.

[0677] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0678] Step 1:

[0679] A user launches an app on their smartphone and starts the voice capture function. When the user starts a phone call or face-to-face conversation, the device starts capturing voice data in real time. The input is the user's voice data, and the output is the captured voice data.

[0680] Step 2:

[0681] The captured voice data is sent to the Google Cloud Speech-to-Text API, where it is converted into text data using speech recognition technology. The input is voice data, and the output is converted text data. This conversion process analyzes the linguistic characteristics of the voice data and the speaker's voice and converts it into text.

[0682] Step 3:

[0683] The converted text data is sent to a server and analyzed by a generative AI model (OpenAI GPT). The input is text data and the output is the analysis result. In this step, the text data is matched with fraud patterns to evaluate the likelihood of fraud or criminal activity. Specifically, the model is queried using a specific prompt sentence.

[0684] Rate the following text as a scam: Your credit card details may have been stolen. Check now

[0685] Step 4:

[0686] The server generates a warning message and countermeasure suggestions based on the analysis results. The input is the analysis results, and the output is a warning message and countermeasure suggestions. If the analysis results indicate a possibility of fraud, a warning message is created and saved as data to be sent to the user.

[0687] Step 5:

[0688] The generated warning message and suggested measures are sent to the user's smartphone using the Twilio API. The input is the warning message and suggested measures, and the output is the warning message displayed on the user's smartphone. Specifically, an SMS is sent via the Twilio API, and a pop-up message is displayed on the user's device.

[0689] Step 6:

[0690] After the conversation is over, all collected data is sent to the server and stored in a database. The input is the conversation data and analysis results, and the output is the data stored in the database. The stored data is used to retrain the generative AI model, contributing to improving the accuracy of the system.

[0691] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0692] This invention is a system that converts voice data into text data in real time, evaluates the possibility of fraud or criminal activity using a generative artificial intelligence model and an emotion engine, and presents a warning and countermeasure suggestions to the user. Specific embodiments are described below.

[0693] System configuration and operation

[0694] 1. The user launches the app

[0695] Users launch a dedicated app on their smartphone or tablet.

[0696] The device will display the app interface and begin voice capture and emotion recognition functions.

[0697] 2. Real-time conversation transcription and emotion recognition

[0698] The user answers a phone call or starts a face-to-face conversation. The device begins capturing audio through the microphone.

[0699] The device converts voice data into text data in real time, while the emotion engine simultaneously recognizes emotions from the user's voice and facial expressions.

[0700] 3. Sending text data and emotion data

[0701] The device then transmits the generated text data and emotion data to a server, where the data is encrypted using a secure communication protocol.

[0702] 4. Analysis of text data and emotion data

[0703] The server inputs the received text data into a generative AI model, while inputting the emotion data into an emotion evaluation engine.

[0704] The generative AI model analyzes the conversation content against a database of fraud and criminal patterns, while an emotion assessment engine simultaneously analyzes the user's emotional state.

[0705] 5. Generating comprehensive evaluation results

[0706] The server integrates the analysis results of the generative AI model and the emotion evaluation engine to comprehensively assess the likelihood of fraud or criminal activity.

[0707] If the emotional data indicates strong negative emotions such as anxiety or fear in the user, the importance of the warning and the priority of the countermeasures will be adjusted.

[0708] 6. Providing warnings and suggested solutions

[0709] The server generates a warning message and a countermeasure proposal and sends them to the terminal.

[0710] The device will display a warning to the user as a pop-up message and suggest appropriate measures, such as "This may be a scam" or "Please check with official contacts."

[0711] 7. Recording speech patterns and emotional data

[0712] When the conversation is finished, the terminal sends all data to the server.

[0713] The server records the received data and stores it in a back-end database, which can be used to retrain future AI models and emotion engines.

[0714] 8. Retraining AI models and emotion engines

[0715] The server periodically retrains the generative AI model and emotion engine with new speech patterns and emotion data, thereby improving the accuracy of the model and engine.

[0716] Specific examples

[0717] Scenario: Fake billing call

[0718] 1. Launch the app

[0719] An elderly user launches a dedicated app.

[0720] The user prepares to receive the call.

[0721] 2. Conversation initiation, transcription, and emotion recognition

[0722] The user receives a call and the conversation begins with the user being told that the outstanding amount is due and that they should pay it immediately.

[0723] The device captures conversations in real time and instantly transcribes them into text, while simultaneously detecting anxiety from the user's tone and content of voice.

[0724] 3. Data submission and analysis

[0725] The terminal transmits the generated text data and emotion data to the server.

[0726] The data received by the server is analyzed using a generative AI model and emotion evaluation engine.

[0727] A generative AI model detects fraud patterns such as "non-payment" and "pay now," while a sentiment assessment engine detects high levels of anxiety in users.

[0728] 4. Comprehensive evaluation and warning message generation

[0729] The server determines that there is a high possibility of fraud and generates a warning message and a suggestion to "check with an official contact." Based on the emotional data, the importance of the warning is increased, urging even more caution.

[0730] 5. Displaying warnings and suggested solutions

[0731] The terminal displays these messages to the user as popups.

[0732] 6. Record data and retrain

[0733] After the conversation ends, the device sends all data to the server.

[0734] The server records the data and uses it to retrain the generative AI model and emotion engine.

[0735] This system not only effectively protects users from fraud and criminal activity, but also suggests countermeasures based on the user's emotional state. This increases users' sense of security and helps prevent fraud from occurring. Furthermore, the accuracy of the system can be constantly improved based on the collected data.

[0736] ---

[0737] The above is an explanation of the "Mode for Carrying Out the Invention" based on the scope of the patent claims for the invention that combines an emotion engine. This content will help you understand the specific implementation methods of the invention and you can use it as a reference to carry out the invention.

[0738] The processing flow will be explained below.

[0739] Step 1:

[0740] The user launches the dedicated app on their smartphone or tablet, which displays the app's interface and starts the voice capture and emotion recognition functions.

[0741] Step 2:

[0742] When a user answers a phone call or starts a face-to-face conversation, the device begins capturing audio through the microphone. Audio capture begins immediately and the captured audio data is converted into text data in real time. At the same time, the emotion engine recognizes emotions from the user's voice and facial expressions in real time.

[0743] Step 3:

[0744] The device then transmits the generated text data and emotion data to a server, where the data is encrypted using a secure communication protocol.

[0745] Step 4:

[0746] The server inputs the received text data into a generative AI model and begins analysis. Meanwhile, it inputs the emotion data into an emotion evaluation engine to evaluate the user's emotional state.

[0747] Step 5:

[0748] The generative AI model analyzes the text data by comparing it with a database of fraud and criminal patterns, while the emotion assessment engine evaluates the user's emotional state, particularly anxiety and fear, and integrates this data into the generative AI model's analysis.

[0749] Step 6:

[0750] The server combines the analysis results of the generated AI model and the emotion evaluation engine to comprehensively evaluate the likelihood of fraud or criminal activity, adjust the importance and content of warning messages based on the emotion data, and generate the final evaluation result.

[0751] Step 7:

[0752] The server generates a warning message and suggested countermeasures and sends them to the device. The device receives the message and displays a pop-up warning to the user, suggesting appropriate countermeasures, such as "This may be a scam" or "Please check with the official contacts."

[0753] Step 8:

[0754] When the user ends the call, the device transmits all conversation data (voice and transcription data) and emotion data to the server, where the data is again encrypted and transmitted via a secure communication protocol.

[0755] Step 9:

[0756] The server records the received data and stores it in a back-end database, which then uses it to retrain the generative AI model and emotion engine.

[0757] Step 10:

[0758] The server periodically retrains the generative AI model and emotion engine based on new speech patterns and emotion data, thereby improving the accuracy of the model and engine.

[0759] ---

[0760] The above is a concrete explanation of the processing steps of the invention that combines the emotion engine. This allows you to clearly understand how the system operates and the specific processing that is performed.

[0761] Example 2

[0762] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0763] Conventional systems could convert voice data into text data in real time to assess the likelihood of fraud or criminal activity, but they were unable to adjust the severity of warnings or the priority of countermeasures while taking the user's emotional state into account. As a result, it was not possible to propose appropriate countermeasures based on the user's emotional state, making it difficult to sufficiently reduce the risk of fraud or criminal activity. The purpose of this invention is to provide an effective fraud and criminal activity assessment system that also takes the user's emotional state into account.

[0764] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing voice data and text data and recognizing the user's emotional state in real time, a means for analyzing the text data using a generative artificial intelligence model and evaluating the possibility of fraud or criminal activity, and a means for integrating the analysis results and the user's emotional state to adjust the importance of the warning and the priority of countermeasures. This makes it possible to more accurately evaluate the possibility of fraud or criminal activity and to provide appropriate warnings and propose countermeasures according to the user's emotional state.

[0765] "Voice data" refers to digital audio information captured from a user's speech via a microphone.

[0766] "Text data" is digital document information that is created by converting voice data into text in real time.

[0767] A "generative artificial intelligence model" is a machine learning algorithm used to analyze text data and understand specific patterns and meanings.

[0768] "Emotional state" refers to the psychological and emotional state of the user, which is analyzed from the user's tone of voice, facial expression, etc.

[0769] "Real-time" refers to operations and processing occurring instantly with minimal delay.

[0770] "Fraud and criminal activity" refers to any activity that involves using dishonest means to deceive others for profit or that violates the law.

[0771] "Evaluation means" refers to techniques or methods for analyzing specific information based on input data and deriving results.

[0772] A "warning message" is a notification message that alerts the user and notifies them of the existence of a risk.

[0773] A "measure proposal" is a proposal that instructs the user on specific actions and methods to take in response to a detected risk.

[0774] "Encryption means" is a technology that encrypts data using a certain algorithm to transmit the data securely, making the contents of the data unintelligible to third parties.

[0775] A "server" is a computer system for storing, analyzing, and providing data.

[0776] A "terminal" is a device (e.g., a smartphone or tablet) that a user directly operates and that communicates with a server.

[0777] A "secure communication protocol" is a standardized procedure or rule for securely communicating data (e.g., HTTPS).

[0778] This invention is a system that converts voice data into text data in real time, uses a generative artificial intelligence model and an emotion evaluation engine to evaluate the possibility of fraud or criminal activity, and presents warnings and suggested countermeasures to the user.

[0779] System configuration and operation

[0780] 1. Launch the app

[0781] The user launches the dedicated app on their smartphone or tablet. The device displays the app's interface and is ready to begin voice capture and emotion recognition. The app has a user-friendly interface, including a button that says "Start Call."

[0782] 2. Audio data capture and conversion

[0783] When a user answers a call, the device's microphone captures the audio data, which is then converted into text data in real time using the Google Cloud Speech-to-Text API, instantly recording what the user is saying.

[0784] 3. Emotion recognition

[0785] At the same time, the system uses the Microsoft Azure Cognitive Services Emotion API to analyze the user's emotions in real time from the tone and content of their voice. This API recognizes the speaker's emotions from voice data and can identify negative emotions such as anxiety or fear.

[0786] 4. Data transmission

[0787] The device encrypts the generated text data and emotion data using a secure communication protocol (e.g., HTTPS) and transmits it to the server, where it is protected from unauthorized access.

[0788] 5. Data Analysis

[0789] The server inputs the received text data into OpenAI's GPT-4 model to evaluate the likelihood of fraud or criminal activity, and analyzes the emotional data using Azure's Emotion API. This allows for highly accurate detection of fraud and criminal activity based on the user's emotional state and text data.

[0790] 6. Generating comprehensive evaluation results

[0791] The server integrates the analysis results of the generative AI model and the emotion evaluation engine to comprehensively evaluate the likelihood of fraud or criminal activity, and adjusts the importance of warnings and the priority of countermeasures based on the level of anxiety or fear expressed by the user as indicated by the emotion data.

[0792] 7. Providing warnings and suggested solutions

[0793] The server generates a warning message and suggested actions and sends it to the device, which then displays these messages to the user as a pop-up notification, such as "This may be a scam" or "Please check with an official contact."

[0794] 8. Record data and retrain

[0795] After the conversation is over, the device sends all data to the server, which stores it in a back-end database. This stored data will be used to retrain the generative AI model and emotion engine in the future. Periodic retraining ensures the system stays up to date with the latest fraud patterns and the user's emotional state.

[0796] Specific examples

[0797] Scenario: Fake billing call

[0798] Launching the app

[0799] The elderly user launches the dedicated app and prepares to receive a call. The device then starts the voice capture function.

[0800] Conversation initiation, transcription, and emotion recognition

[0801] The user receives a call and the conversation begins with the message, "Your payment is outstanding and you need to pay it immediately." The device captures the conversation in real time and instantly transcribes it into text using the Google Cloud Speech-to-Text API. At the same time, it uses Azure's Emotion API to detect anxiety from the user's tone and content of voice.

[0802] Data transmission and analysis

[0803] The device sends the generated text data and emotion data to a server, which then analyzes the received data using OpenAI's GPT-4 model and Azure Emotion API. The GPT-4 model detects fraudulent patterns such as "unpaid" and "pay immediately," while the emotion evaluation engine detects high levels of anxiety in the user.

[0804] Evaluation and warning message generation

[0805] The server determines that there is a high possibility of fraud and generates a warning message and a suggestion to "check with an official contact." Based on the emotional data, the importance of the warning is increased, urging even more caution.

[0806] Displaying warnings and suggested solutions

[0807] The terminal displays these messages to the user as pop-up notifications.

[0808] Recording Data and Retraining

[0809] After the conversation ends, the device sends all data to the server, which records it and uses it to retrain the generative AI model and emotion engine.

[0810] Prompt Sentence Examples

[0811] What should you do if you receive a call saying, "Please transfer the money right now"?

[0812] By inputting this prompt into a generative AI model, the system can suggest appropriate countermeasures.

[0813] The above is an embodiment of the present invention. This system effectively protects users from fraud and criminal activity and provides countermeasure suggestions according to the user's emotional state.

[0814] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0815] Step 1:

[0816] A user launches the dedicated app on a smartphone or tablet. The device displays the app's interface and prepares to start the voice capture and emotion recognition functions. The input is the user's operation, and the output is the display of the app's home screen, which includes a "Start phone call" button.

[0817] Step 2:

[0818] When a user receives a call, the device's microphone captures audio data. The input is the captured audio data, and the output is a real-time stream of that audio data. The device then sends the captured audio data to the Google Cloud Speech-to-Text API, which converts it into text data. Specifically, the audio data is analyzed in real time and instantly converted into text.

[0819] Step 3:

[0820] At the same time, the device uses the Emotion API from Microsoft Azure Cognitive Services to analyze the user's emotions in real time. The input is the captured voice data, and the output is the emotion recognition results. Specifically, the Emotion API identifies negative emotions from the tone and content of the voice.

[0821] Step 4:

[0822] The text data and emotion data generated by the device are encrypted using a secure communication protocol (e.g., HTTPS) and sent to the server. The input is text data and emotion data, and the output is the encrypted data. This ensures data security.

[0823] Step 5:

[0824] The server inputs the received text data into OpenAI's GPT-4 model to evaluate the likelihood of fraud or criminal activity. The input is the received text data, and the output is the analysis result. Specifically, the GPT-4 model detects keywords such as "unpaid" and "pay immediately" in the text.

[0825] Step 6:

[0826] The server analyzes the emotion data using an emotion evaluation engine. The input is the received emotion data, and the output is the analyzed emotion evaluation result. Specifically, the Emotion API evaluates the user's anxiety or fear level.

[0827] Step 7:

[0828] The server integrates the analysis results of the generative AI model and the emotion evaluation engine to generate a comprehensive evaluation result. The input is the analysis results of the text data and the evaluation results of the emotion data, and the output is a comprehensive evaluation report. Specifically, the server combines both sets of data to evaluate the likelihood and severity of fraud or criminal activity.

[0829] Step 8:

[0830] The server generates a warning message and suggested countermeasures based on the evaluation results and sends them to the terminal. The input is the overall evaluation result, and the output is a warning message and suggested countermeasures. Specifically, the server generates messages such as "This may be a scam" or "Please check with the official contact point."

[0831] Step 9:

[0832] The terminal displays the received warning message and suggested countermeasures to the user as a pop-up notification. The input is the warning message and suggested countermeasures sent from the server, and the output is the notification to the user. Specifically, the terminal displays a pop-up message on the screen to immediately notify the user of the warning.

[0833] Step 10:

[0834] After the conversation is over, the device sends all data to the server. The input is the generated text data and emotion data, and the output is data sent to the server. The server stores this data in a backend database. The stored data is used to retrain the generative AI model and emotion engine. This improves the accuracy of the system and allows it to adapt to the latest fraud patterns and the user's emotional state.

[0835] (Application example 2)

[0836] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0837] Fraud and criminal activity methods are evolving daily, resulting in a constant stream of fraud victims perpetrated over the phone or in face-to-face conversations. Elderly people and those with low digital literacy are particularly susceptible to scams, and are prone to financial loss and psychological damage. However, conventional fraud prevention systems struggle to provide real-time countermeasures and are unable to provide warnings or suggest countermeasures that take into account the user's emotional state. To solve this problem, a system is needed that converts speech into text in real time, uses a generative artificial intelligence model and an emotion engine to comprehensively evaluate the likelihood of fraud or criminal activity, and then provides appropriate warnings and suggests countermeasures to the user.

[0838] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting voice data into text data in real time, means for analyzing the text data using a generative artificial intelligence model and evaluating the possibility of fraud or criminal activity, and means for recognizing the user's emotional state using an emotion engine and displaying a warning message and suggested countermeasures to the user based on the evaluation results. This makes it possible to detect fraud or criminal activity in real time and issue appropriate warnings and suggest countermeasures according to the user's emotional state.

[0839] "Audio data" refers to audio signals uttered by a user recorded as digital information.

[0840] "Text data" is a collection of character information generated by analyzing voice data.

[0841] A "generative artificial intelligence model" is an algorithm that uses machine learning to analyze text data and understand and evaluate specific patterns and meanings.

[0842] The "emotion engine" is an algorithm that analyzes the user's emotional state from their voice and facial expressions and evaluates the results.

[0843] A "warning message" is an informational notice intended to inform the user of a high possibility of fraud or criminal activity.

[0844] "Measures Proposal" is an information notification that indicates specific actions or measures that the user should take based on the warning message.

[0845] A "server" is a computer system that analyzes voice data and text data and generates comprehensive evaluation results.

[0846] A "terminal" is a device used by a user, which has the function of capturing voice data and displaying warning messages and suggested countermeasures received from the server.

[0847] This invention is a system that converts voice data into text data in real time, evaluates the possibility of fraud or criminal activity using a generative AI model and an emotion engine, and presents warnings and countermeasure suggestions to users. This system is mainly composed of a server, a terminal, and a user.

[0848] System configuration

[0849] Terminal

[0850] The terminal is a device such as a smartphone or tablet that has a microphone for capturing audio data and a display for displaying text data and warning messages.

[0851] When a user initiates a phone or face-to-face conversation, the device captures voice data and converts it into text data in real time using a speech recognition engine.

[0852] server

[0853] The server is a central computer system that receives and analyzes the text data and emotion data sent from the terminals.

[0854] To analyze the text data, a generative AI model (such as BERT or GPT) is used. The generative AI model compares patterns of fraud and criminal activity with a database to assess whether the conversation is fraudulent. In addition, an emotion engine analyzes the user's emotional state based on the voice data sent from the device. This makes it possible to determine whether the user is feeling anxious or fearful.

[0855] User

[0856] A user is an individual who uses the system and is the sender of voice data. When a user answers a phone call or has a face-to-face conversation, the voice data is captured through the terminal and sent to the server. Analysis results and warning messages from the server are notified to the user through the terminal.

[0857] Program processing

[0858] 1. Capture and convert voice data: Capture voice using the device's microphone and convert it into text data in real time using the Google Speech Recognition API.

[0859] 2. Data Analysis: Send the text data and emotion data to the server, where the text data is analyzed using a generative AI model (e.g., BERT), and the emotion engine is used to analyze the user's emotional state.

[0860] 3. Evaluation and Notification: The server integrates the results of analysis by the generative AI model and emotion engine, and if there is a high possibility of fraud, a warning message is generated and sent to the device. The notification content may be something like "This is likely a scam" or "Please check with the official contact point."

[0861] Specific examples

[0862] Scenario: Fake billing call

[0863] 1. User receives a call: The elderly user receives a call and launches the app on their device.

[0864] 2. Real-time transcription of the conversation: The audio data of "It's unpaid and please pay it immediately" is captured and converted into text data in real time.

[0865] 3. Data transmission and analysis: Text data and emotion data are transmitted to the server, and the generative AI model detects fraud patterns such as "unpaid" and "pay immediately." At the same time, the emotion engine detects high levels of anxiety from the user.

[0866] 4. Integration of evaluation results and generation of warning: The server determines that there is a high possibility of fraud and generates a warning message such as "Please check with the official contact point."

[0867] 5. Warning Notifications: The device will display these messages to the user as a popup.

[0868] Prompt Sentence Examples

[0869] "Pay any outstanding amounts immediately."

[0870] Category labels: Scam, Safety

[0871] In this way, the system can provide users with real-time warnings and countermeasures against fraud and criminal activity, helping to keep them safe.

[0872] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0873] Step 1:

[0874] A user launches an app on a smartphone or tablet and starts a phone or face-to-face conversation. The device's microphone captures audio data. The input is the user's voice data, which is recorded as a digital audio signal by the app's microphone. The output is the captured voice data.

[0875] Step 2:

[0876] The device converts captured voice data into text data in real time. This conversion is performed using a speech recognition engine (e.g., Google Speech Recognition API). The input is the captured voice data, and the output is the converted text data.

[0877] Step 3:

[0878] The converted text data and emotion data detected from the user's voice tone and content are sent from the device to the server. The data is encrypted and sent using a secure communication protocol (e.g., SSL / TLS). The input is text data and emotion data, and the output is data that is sent securely.

[0879] Step 4:

[0880] The server analyzes the received text data using a generative AI model (e.g., BERT). This analysis evaluates the text data to detect patterns of fraud and criminal activity. The input is the submitted text data, and the output is the generative AI model's evaluation of the fraud and criminal activity.

[0881] Step 5:

[0882] In parallel, the server inputs the received emotion data into the emotion engine to analyze the user's emotional state. The emotion engine evaluates the user's negative emotions, such as anxiety, discomfort, and fear, from the voice data and text. The input is the transmitted emotion data, and the output is the analysis result of the user's emotional state.

[0883] Step 6:

[0884] The server integrates the evaluation results of the generative AI model and the analysis results of the emotion engine to comprehensively determine the likelihood of fraud or criminal activity. Depending on the evaluation results, a warning message and countermeasure proposals are generated. The input is the analysis results of text data and emotion data, and the output is the integrated evaluation results and the warning message and countermeasure proposals based on them.

[0885] Step 7:

[0886] The server generates a warning message and sends it to the terminal. The input is the warning message and the countermeasure proposal, and the output is the notification data that is securely transmitted.

[0887] Step 8:

[0888] The terminal displays a warning message and suggested countermeasures to the user as a pop-up message. This allows the user to receive warnings about possible fraud or criminal activity and appropriate countermeasures in real time. The input is the warning message and suggested countermeasures sent from the server, and the output is the pop-up message displayed on the terminal display.

[0889] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0890] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0891] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0892] [Third embodiment]

[0893] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0894] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0895] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0896] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0897] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0898] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0899] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0900] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0901] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0902] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0903] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0904] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0905]

[0906] This invention is a system that converts voice data into text data in real time, evaluates the possibility of fraud or criminal activity using a generative artificial intelligence model, and presents a warning and countermeasure suggestions to the user. Specific embodiments will be described below.

[0907] System configuration and operation

[0908] 1. The user launches the app

[0909] Users launch a dedicated app on their smartphone or tablet.

[0910] The device will display the app interface and start the audio capture function.

[0911] When a user starts a phone or in-person conversation, the app automatically begins capturing audio.

[0912] 2. Real-time transcription of conversations

[0913] The device converts the captured audio into text data in real time, using voice recognition technology to instantly convert spoken words into written text.

[0914] Once the transcription is complete, the text data is sent to the server.

[0915] 3. Analysis and Evaluation of Text Data

[0916] The server inputs the received text data into a generative artificial intelligence model.

[0917] The generative AI model analyzes the content of the conversation by comparing it with a database of patterns of fraud and criminal activity.

[0918] If fraud or criminal activity is suspected, the server evaluates the results and generates a warning message.

[0919] 4. Present warnings and suggested solutions

[0920] The server returns data including a warning message and suggested countermeasures to the terminal.

[0921] The device will warn the user with a pop-up message and suggest appropriate measures, such as "This may be a scam" or "Please check with official contacts."

[0922] 5. Recording conversation patterns

[0923] The terminal sends the data of the completed conversation to the server, where it is recorded in the backend.

[0924] The server analyzes the stored data and uses it as data for retraining the generative AI model.

[0925] Specific examples

[0926] Scenario: Fake billing call

[0927] 1. Launch the app

[0928] An elderly user launches a dedicated app.

[0929] The user prepares to receive the call.

[0930] 2. Start a conversation and transcribe

[0931] The user receives a call and the conversation begins with the user being told that the outstanding amount is due and that they should pay it immediately.

[0932] The device captures the conversation in real time and instantly transcribes it into text.

[0933] 3. Sending and analyzing text data

[0934] The terminal transmits the generated text data to the server.

[0935] The data received by the server is analyzed using a generative AI model to detect fraudulent patterns such as "unpaid" and "pay immediately."

[0936] 4. Displaying warnings and suggested solutions

[0937] The server determines that there is a high possibility of fraud and generates a warning message and a suggestion to "check with the official contact point."

[0938] The terminal displays these messages to the user as popups.

[0939] 5. Recording conversation patterns

[0940] After the conversation is over, the device sends all data to the server.

[0941] The server records the data and uses it to retrain the generative AI model.

[0942] This system not only effectively protects users from fraud and criminal activity, but also allows the accuracy of the generative AI model to be constantly improved based on the collected data.The system covers both online and offline risks, making it effective for a wide range of users.

[0943] ---

[0944] The above is the explanation of the "Mode for Carrying Out the Invention" based on the scope of the claims. This content will help you understand the specific methods for carrying out the invention and you can use it as a reference to carry out the invention.

[0945] The processing flow will be explained below.

[0946] Step 1:

[0947] The user launches the dedicated app on their smartphone or tablet. The device displays the app interface and starts the audio capture function.

[0948] Step 2:

[0949] The user answers a phone call or starts a face-to-face conversation. The device begins capturing audio through the microphone. Once audio capture begins, the device transcribes the captured audio data in real time.

[0950] Step 3:

[0951] The device generates the transcribed text data and sends it to the server, where it is encrypted using a secure communication protocol.

[0952] Step 4:

[0953] The server inputs the received text data into a generative AI model, which then compares it with a database of fraud and criminal patterns and analyzes the conversation.

[0954] Step 5:

[0955] The generative AI model returns the analysis results to the server, which evaluates them and determines whether there is any potential for fraud or criminal activity.

[0956] Step 6:

[0957] The server generates a warning message based on the evaluation results, including specific warning content and suggested countermeasures for the user.

[0958] Step 7:

[0959] The server sends back data containing a warning message and suggested solutions to the terminal, which receives it and displays it to the user as a pop-up message.

[0960] Step 8:

[0961] The user ends the call. The device sends all conversation data (audio and transcription data) to the server.

[0962] Step 9:

[0963] The server records the data it receives and stores it in a back-end database, which can then be used to retrain the AI ​​model in the future.

[0964] Step 10:

[0965] The server periodically retrains the generative AI model based on new conversation patterns, thereby improving the model's detection accuracy.

[0966] ---

[0967] This is a detailed explanation of the program's processing steps, which will give you a clear understanding of how the system works and what specific processes are performed.

[0968] Example 1

[0969] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0970] Conventional methods have had difficulty detecting fraud and criminal activity in real time and quickly warning users. They also lacked the means to securely transmit voice data and properly return analysis results and warning messages to the device. Therefore, there was a need for an effective means for users to prevent themselves from becoming victims of fraud and criminal activity.

[0971] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0972] In this invention, the server includes means for converting voice data into text data in real time, means for analyzing the text data using a generative artificial intelligence model to evaluate the likelihood of fraud or criminal activity, means for displaying a warning message and suggested countermeasures to the user based on the evaluation results, means for transmitting text data transmitted from the user to the terminal to the server, means for analyzing the text data using the generative artificial intelligence model to detect the likelihood of fraud or criminal activity, means for returning the generated warning message and suggested countermeasures to the terminal, and means for recording the conversation data and using it as data for retraining, thereby enabling users to be effectively protected from fraud and criminal activity.

[0973] "Audio data" means data in digital form that contains human speech or other sound information.

[0974] "Text data" is data consisting of characters and symbols that is obtained as a result of converting voice data.

[0975] A "generative artificial intelligence model" is an artificial intelligence model with natural language processing capabilities that is trained based on a large dataset.

[0976] "Analysis" is the process performed to determine the possibility of fraud or criminal activity based on the input data.

[0977] A "warning message" is a message that is generated as a result of analysis to warn or alert the user.

[0978] "Measure suggestions" are specific actions or measures recommended to the user based on the warning message.

[0979] "User" refers to any individual or corporation that uses this system.

[0980] "Terminal" refers to an electronic device, such as a smartphone or tablet, that allows a user to access and use the system.

[0981] A "server" is a central computer system used to process voice and text data and provide information to users.

[0982] "Real-time" refers to processing that occurs almost simultaneously with the occurrence of an event.

[0983] "Conversation data" refers to data that includes captured speech and text data generated based on the speech.

[0984] "Retraining" is the process of retraining a generative artificial intelligence model using new data to improve its accuracy.

[0985] MODE FOR CARRYING OUT THE INVENTION

[0986] The present invention provides a system that converts voice data into text data in real time, evaluates the possibility of fraud or criminal activity using a generative artificial intelligence model, and presents a warning and countermeasure suggestions to the user. A specific embodiment of the present invention will be described below.

[0987] System configuration and operation

[0988] The user launches the app

[0989] Users launch a dedicated app on their smartphone or tablet. This app has a voice capture function, and when the user starts a phone or face-to-face conversation, the device begins capturing the voice in real time.

[0990] Voice capture to text conversion

[0991] The device converts the captured voice data into text data in real time using voice recognition software (e.g., Google Speech-to-Text API), and the converted text data is immediately sent to the server.

[0992] Text data analysis

[0993] The server inputs the received text data into a generative AI model (e.g., OpenAI GPT-4). The generative AI model analyzes the conversation by comparing it with a database of fraudulent and criminal patterns. If the analysis reveals a possibility of fraud or criminal activity, the server generates a warning message and suggests countermeasures.

[0994] Warnings and suggested solutions

[0995] The server generates a warning message and sends it back to the device, which displays it to the user as a pop-up message. Specific messages include "This may be a scam" and "Please check with your official contacts."

[0996] Recording and retraining conversation data

[0997] After the conversation is complete, the device sends all conversation data to the server, which records it in a database. This data is used to retrain the generative AI model, helping to improve the accuracy of the system.

[0998] Specific operation example

[0999] Scenario: Fake billing call

[1000] 1. Launch the app

[1001] An elderly user launches a dedicated app.

[1002] The user prepares to receive the call.

[1003] 2. Start a conversation and transcribe

[1004] The user receives a call and the conversation begins with the user being told that the outstanding amount is due and that they should pay it immediately.

[1005] The device captures the conversation in real time and instantly transcribes it into text.

[1006] 3. Sending and analyzing text data

[1007] The terminal transmits the generated text data to the server.

[1008] The data received by the server is analyzed using a generative artificial intelligence model to detect fraudulent patterns such as "non-payment" and "immediate payment."

[1009] 4. Displaying warnings and suggested solutions

[1010] The server determines that there is a high possibility of fraud and generates a warning message and a suggestion to "check with the official contact point."

[1011] The terminal displays these messages to the user as popups.

[1012] 5. Recording conversation patterns

[1013] After the conversation is over, the terminal sends all data to the server.

[1014] The server records the data and uses it to retrain the generative AI model.

[1015] In this way, the system of the present invention can consistently convert voice data in real time, analyze it using a generative AI model, and promptly warn users and suggest countermeasures. This system effectively protects users from fraud and criminal activity, and the accuracy of the system is continuously improved.

[1016] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1017] Step 1:

[1018] The user launches the app

[1019] Input: The user operates a smartphone or tablet device to launch a dedicated app.

[1020] What it does: The device displays the app interface and puts the audio capture feature into standby.

[1021] Output: The audio capture feature is enabled and audio is recorded when the user starts speaking.

[1022] Step 2:

[1023] The device captures audio in real time

[1024] Input: Ambient sounds, especially phone calls and face-to-face conversations you have.

[1025] What it does: Your device uses the app's audio capture feature to record audio data in real time.

[1026] Output: Recorded audio data.

[1027] Step 3:

[1028] The device converts the voice data into text data.

[1029] Input: Real-time recorded audio data.

[1030] How it works: Your device uses speech recognition software (e.g., Google Speech-to-Text API) to convert voice data into text.

[1031] Output: The converted text data.

[1032] Step 4:

[1033] The device sends text data to the server.

[1034] Input: The converted text data.

[1035] How it works: The device sends text data to the server using a secure communication protocol (e.g., HTTPS).

[1036] Output: Text data is sent to the server.

[1037] Step 5:

[1038] The server analyzes the text data using a generative AI model

[1039] Input: The text data sent to the server.

[1040] How it works: The server feeds text data into a generative artificial intelligence model (e.g., OpenAI GPT-4) to analyze it for potential fraud or criminal activity.

[1041] Output: Analysis results in assessment data of the likelihood of fraud or criminal activity.

[1042] Step 6:

[1043] The server evaluates the results and generates a warning message

[1044] Input: Analysis results generated by a generative AI model.

[1045] How it works: The server evaluates the analysis results and generates a warning message and suggested actions if there is a high possibility of fraud or criminal activity.

[1046] Output: Generated warning messages and suggested actions.

[1047] Step 7:

[1048] The server sends a warning message and suggested solutions back to the device.

[1049] Input: Generated warning message and suggested action.

[1050] Operation: The server returns a warning message and suggested solutions to the device as an HTTP response.

[1051] Output: A warning message and suggested solutions will be sent to your device.

[1052] Step 8:

[1053] The device displays a warning and suggests countermeasures to the user.

[1054] Input: The warning message and suggested action returned by the server.

[1055] How it works: The device displays a pop-up notification to the user with a warning message and suggested solutions.

[1056] Output: User confirms warning message and suggested action.

[1057] Step 9:

[1058] The device records conversation patterns and sends them to the server.

[1059] Input: Finished conversation data.

[1060] Operation: The terminal records the conversation data that has been completed and sends it to the server.

[1061] Output: Conversation data recorded on the server.

[1062] Step 10:

[1063] The server records the data and uses it to retrain the generative AI model.

[1064] Input: Conversation data sent to the server.

[1065] How it works: The server stores conversation data and uses it as retraining data for generative AI models.

[1066] Output: An updated generative AI model and conversation data recorded in a database.

[1067] (Application example 1)

[1068] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1069] In modern society, fraud and criminal acts are becoming increasingly sophisticated, and many people are falling victim to these methods. To address this situation, a system is needed that can detect fraud and criminal acts early and provide appropriate warnings and countermeasures. However, current technology does not provide a system that can analyze voice data in real time and immediately evaluate the possibility of fraud and provide a warning. Therefore, an objective of the present invention is to provide a system that can detect fraud and criminal acts in real time and provide a warning, allowing users to communicate safely.

[1070] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1071] In this invention, the server includes means for converting voice data into text data in real time, means for analyzing the text data using a generative artificial intelligence model to evaluate the possibility of fraud or criminal activity, means for displaying a warning message and countermeasure suggestions to the user based on the evaluation results, means for providing a voice capture function, means for converting voice into text using voice recognition technology, means for analyzing the text by matching it with fraud patterns, and means for sending a warning message to the user using communication means, thereby enabling the user to be aware of the possibility of fraud or criminal activity in real time and take appropriate countermeasures.

[1072] "Audio data" refers to continuous sound signals obtained from conversations, communications, etc.

[1073] "Text data" is voice data converted into a string of characters in a language.

[1074] A "generative artificial intelligence model" is an AI technology that learns large amounts of data and generates and analyzes sentences and information.

[1075] "Voice recognition technology" is a technology that converts voice data into text in real time.

[1076] "Fraud patterns" are data that indicate combinations of words and phrases associated with fraud.

[1077] A "warning message" is a notification that alerts the user to danger or problems.

[1078] "Measure suggestions" are advice that shows specific actions or methods that users should take.

[1079] "User" refers to any person or organization that uses this system.

[1080] The "audio capture function" is a function for collecting audio in real time.

[1081] "Communication means" refers to the technology and protocols used to exchange information.

[1082] A "server" is a computer system that processes and stores data and provides services over a network.

[1083] "Terminal" refers to a device or equipment that is directly operated by a user.

[1084] "Analysis results" refers to the results of analysis using a generative artificial intelligence model.

[1085] MODE FOR CARRYING OUT THE INVENTION

[1086] This invention is a security system for users that converts voice data into text data in real time, analyzes the text data using a generative artificial intelligence model to evaluate the possibility of fraud or criminal activity, and provides the user with a warning and suggestions for countermeasures. This system is mainly composed of the following elements.

[1087] 1. Hardware and Software Selection

[1088] The following hardware and software are used to realize this system.

[1089] Smartphone terminal: A device operated by a user.

[1090] Microphone: Used to capture audio data.

[1091] Server: Processes data and manages the database.

[1092] Google Cloud Speech-to-Text API: Provides speech recognition technology.

[1093] Generative AI models (e.g., OpenAI GPT): Analyze text data to assess the likelihood of fraud or criminal activity.

[1094] Twilio API: A communication method for sending alert messages to users.

[1095] 2. Explanation of program processing

[1096] The main processing flow of the program of this system is explained below.

[1097] User voice input

[1098] The user starts a conversation using a smartphone. The device's voice capture function is activated, capturing the conversation's audio data in real time. This allows the user's conversation to be input into the system in real time and prepared for analysis.

[1099] Converting audio data to text

[1100] The server converts the captured audio data into text data using the Google Cloud Speech-to-Text API, which then sends the audio data to the server as text data.

[1101] Text data analysis

[1102] The server uses a generative AI model (OpenAI GPT) to analyze the converted text data. This identifies patterns of fraud and criminal activity and assesses risk. This analysis process is performed by using specific keywords and phrases as prompts. The following are example prompts:

[1103] Rate the following text as a scam: Your credit card details may have been stolen. Check now

[1104] Warning messages and suggested solutions

[1105] If the server determines based on the analysis results that there is a high possibility of fraud or criminal activity, it uses the Twilio API to send a warning message to the user's device. The warning message also includes specific suggestions for countermeasures. For example, a message such as "This may be a scam. Please contact an official contact point" may be displayed.

[1106] Adding specific examples

[1107] As a concrete example, let's consider a scenario in which an elderly user receives a fraudulent phone call. When the user receives a call stating, "You have an outstanding payment and need to pay it immediately," the audio is captured in real time and converted to text using the Google Cloud Speech-to-Text API. The OpenAI GPT model then evaluates the likelihood of fraud and, if it determines there is a high risk, sends a warning message to the user's smartphone via the Twilio API stating, "There is a high possibility of fraud. Please check with your official contact." In this way, the user can quickly respond to the risk of fraud.

[1108] This system will enable users to receive real-time warnings about fraud and criminal activity, which is expected to help prevent people from falling victim to fraud.

[1109] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1110] Step 1:

[1111] A user launches an app on their smartphone and starts the voice capture function. When the user starts a phone call or face-to-face conversation, the device starts capturing voice data in real time. The input is the user's voice data, and the output is the captured voice data.

[1112] Step 2:

[1113] The captured voice data is sent to the Google Cloud Speech-to-Text API, where it is converted into text data using speech recognition technology. The input is voice data, and the output is converted text data. This conversion process analyzes the linguistic characteristics of the voice data and the speaker's voice and converts it into text.

[1114] Step 3:

[1115] The converted text data is sent to a server and analyzed by a generative AI model (OpenAI GPT). The input is text data and the output is the analysis result. In this step, the text data is matched with fraud patterns to evaluate the likelihood of fraud or criminal activity. Specifically, the model is queried using a specific prompt sentence.

[1116] Rate the following text as a scam: Your credit card details may have been stolen. Check now

[1117] Step 4:

[1118] The server generates a warning message and countermeasure suggestions based on the analysis results. The input is the analysis results, and the output is a warning message and countermeasure suggestions. If the analysis results indicate a possibility of fraud, a warning message is created and saved as data to be sent to the user.

[1119] Step 5:

[1120] The generated warning message and suggested measures are sent to the user's smartphone using the Twilio API. The input is the warning message and suggested measures, and the output is the warning message displayed on the user's smartphone. Specifically, an SMS is sent via the Twilio API, and a pop-up message is displayed on the user's device.

[1121] Step 6:

[1122] After the conversation is over, all collected data is sent to the server and stored in a database. The input is the conversation data and analysis results, and the output is the data stored in the database. The stored data is used to retrain the generative AI model, contributing to improving the accuracy of the system.

[1123] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1124] This invention is a system that converts voice data into text data in real time, evaluates the possibility of fraud or criminal activity using a generative artificial intelligence model and an emotion engine, and presents a warning and countermeasure suggestions to the user. Specific embodiments are described below.

[1125] System configuration and operation

[1126] 1. The user launches the app

[1127] Users launch a dedicated app on their smartphone or tablet.

[1128] The device will display the app interface and begin voice capture and emotion recognition functions.

[1129] 2. Real-time conversation transcription and emotion recognition

[1130] The user answers a phone call or starts a face-to-face conversation. The device begins capturing audio through the microphone.

[1131] The device converts voice data into text data in real time, while the emotion engine simultaneously recognizes emotions from the user's voice and facial expressions.

[1132] 3. Sending text data and emotion data

[1133] The device then transmits the generated text data and emotion data to a server, where the data is encrypted using a secure communication protocol.

[1134] 4. Analysis of text data and emotion data

[1135] The server inputs the received text data into a generative AI model, while inputting the emotion data into an emotion evaluation engine.

[1136] The generative AI model analyzes the conversation content against a database of fraud and criminal patterns, while an emotion assessment engine simultaneously analyzes the user's emotional state.

[1137] 5. Generating comprehensive evaluation results

[1138] The server integrates the analysis results of the generative AI model and the emotion evaluation engine to comprehensively assess the likelihood of fraud or criminal activity.

[1139] If the emotional data indicates strong negative emotions such as anxiety or fear in the user, the importance of the warning and the priority of the countermeasures will be adjusted.

[1140] 6. Providing warnings and suggested solutions

[1141] The server generates a warning message and a countermeasure proposal and sends them to the terminal.

[1142] The device will display a warning to the user as a pop-up message and suggest appropriate measures, such as "This may be a scam" or "Please check with official contacts."

[1143] 7. Recording speech patterns and emotional data

[1144] When the conversation is finished, the terminal sends all data to the server.

[1145] The server records the received data and stores it in a back-end database, which can be used to retrain future AI models and emotion engines.

[1146] 8. Retraining AI models and emotion engines

[1147] The server periodically retrains the generative AI model and emotion engine with new speech patterns and emotion data, thereby improving the accuracy of the model and engine.

[1148] Specific examples

[1149] Scenario: Fake billing call

[1150] 1. Launch the app

[1151] An elderly user launches a dedicated app.

[1152] The user prepares to receive the call.

[1153] 2. Conversation initiation, transcription, and emotion recognition

[1154] The user receives a call and the conversation begins with the user being told that the outstanding amount is due and that they should pay it immediately.

[1155] The device captures conversations in real time and instantly transcribes them into text, while simultaneously detecting anxiety from the user's tone and content of voice.

[1156] 3. Data submission and analysis

[1157] The terminal transmits the generated text data and emotion data to the server.

[1158] The data received by the server is analyzed using a generative AI model and emotion evaluation engine.

[1159] A generative AI model detects fraud patterns such as "non-payment" and "pay now," while a sentiment assessment engine detects high levels of anxiety in users.

[1160] 4. Comprehensive evaluation and warning message generation

[1161] The server determines that there is a high possibility of fraud and generates a warning message and a suggestion to "check with an official contact." Based on the emotional data, the importance of the warning is increased, urging even more caution.

[1162] 5. Displaying warnings and suggested solutions

[1163] The terminal displays these messages to the user as popups.

[1164] 6. Record data and retrain

[1165] After the conversation ends, the device sends all data to the server.

[1166] The server records the data and uses it to retrain the generative AI model and emotion engine.

[1167] This system not only effectively protects users from fraud and criminal activity, but also suggests countermeasures based on the user's emotional state. This increases users' sense of security and helps prevent fraud from occurring. Furthermore, the accuracy of the system can be constantly improved based on the collected data.

[1168] ---

[1169] The above is an explanation of the "Mode for Carrying Out the Invention" based on the scope of the patent claims for the invention that combines an emotion engine. This content will help you understand the specific implementation methods of the invention and you can use it as a reference to carry out the invention.

[1170] The processing flow will be explained below.

[1171] Step 1:

[1172] The user launches the dedicated app on their smartphone or tablet, which displays the app's interface and starts the voice capture and emotion recognition functions.

[1173] Step 2:

[1174] When a user answers a phone call or starts a face-to-face conversation, the device begins capturing audio through the microphone. Audio capture begins immediately and the captured audio data is converted into text data in real time. At the same time, the emotion engine recognizes emotions from the user's voice and facial expressions in real time.

[1175] Step 3:

[1176] The device then transmits the generated text data and emotion data to a server, where the data is encrypted using a secure communication protocol.

[1177] Step 4:

[1178] The server inputs the received text data into a generative AI model and begins analysis. Meanwhile, it inputs the emotion data into an emotion evaluation engine to evaluate the user's emotional state.

[1179] Step 5:

[1180] The generative AI model analyzes the text data by comparing it with a database of fraud and criminal patterns, while the emotion assessment engine evaluates the user's emotional state, particularly anxiety and fear, and integrates this data into the generative AI model's analysis.

[1181] Step 6:

[1182] The server combines the analysis results of the generated AI model and the emotion evaluation engine to comprehensively evaluate the likelihood of fraud or criminal activity, adjust the importance and content of warning messages based on the emotion data, and generate the final evaluation result.

[1183] Step 7:

[1184] The server generates a warning message and suggested countermeasures and sends them to the device. The device receives the message and displays a pop-up warning to the user, suggesting appropriate countermeasures, such as "This may be a scam" or "Please check with the official contacts."

[1185] Step 8:

[1186] When the user ends the call, the device transmits all conversation data (voice and transcription data) and emotion data to the server, where the data is again encrypted and transmitted via a secure communication protocol.

[1187] Step 9:

[1188] The server records the received data and stores it in a back-end database, which then uses it to retrain the generative AI model and emotion engine.

[1189] Step 10:

[1190] The server periodically retrains the generative AI model and emotion engine based on new speech patterns and emotion data, thereby improving the accuracy of the model and engine.

[1191] ---

[1192] The above is a concrete explanation of the processing steps of the invention that combines the emotion engine. This allows you to clearly understand how the system operates and the specific processing that is performed.

[1193] Example 2

[1194] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1195] Conventional systems could convert voice data into text data in real time to assess the likelihood of fraud or criminal activity, but they were unable to adjust the severity of warnings or the priority of countermeasures while taking the user's emotional state into account. As a result, it was not possible to propose appropriate countermeasures based on the user's emotional state, making it difficult to sufficiently reduce the risk of fraud or criminal activity. The purpose of this invention is to provide an effective fraud and criminal activity assessment system that also takes the user's emotional state into account.

[1196] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing voice data and text data and recognizing the user's emotional state in real time, a means for analyzing the text data using a generative artificial intelligence model and evaluating the possibility of fraud or criminal activity, and a means for integrating the analysis results and the user's emotional state to adjust the importance of the warning and the priority of countermeasures. This makes it possible to more accurately evaluate the possibility of fraud or criminal activity and to provide appropriate warnings and propose countermeasures according to the user's emotional state.

[1197] "Voice data" refers to digital audio information captured from a user's speech via a microphone.

[1198] "Text data" is digital document information that is created by converting voice data into text in real time.

[1199] A "generative artificial intelligence model" is a machine learning algorithm used to analyze text data and understand specific patterns and meanings.

[1200] "Emotional state" refers to the psychological and emotional state of the user, which is analyzed from the user's tone of voice, facial expression, etc.

[1201] "Real-time" refers to operations and processing occurring instantly with minimal delay.

[1202] "Fraud and criminal activity" refers to any activity that involves using dishonest means to deceive others for profit or that violates the law.

[1203] "Evaluation means" refers to techniques or methods for analyzing specific information based on input data and deriving results.

[1204] A "warning message" is a notification message that alerts the user and notifies them of the existence of a risk.

[1205] A "measure proposal" is a proposal that instructs the user on specific actions and methods to take in response to a detected risk.

[1206] "Encryption means" is a technology that encrypts data using a certain algorithm to transmit the data securely, making the contents of the data unintelligible to third parties.

[1207] A "server" is a computer system for storing, analyzing, and providing data.

[1208] A "terminal" is a device (e.g., a smartphone or tablet) that a user directly operates and that communicates with a server.

[1209] A "secure communication protocol" is a standardized procedure or rule for securely communicating data (e.g., HTTPS).

[1210] This invention is a system that converts voice data into text data in real time, uses a generative artificial intelligence model and an emotion evaluation engine to evaluate the possibility of fraud or criminal activity, and presents warnings and suggested countermeasures to the user.

[1211] System configuration and operation

[1212] 1. Launch the app

[1213] The user launches the dedicated app on their smartphone or tablet. The device displays the app's interface and is ready to begin voice capture and emotion recognition. The app has a user-friendly interface, including a button that says "Start Call."

[1214] 2. Audio data capture and conversion

[1215] When a user answers a call, the device's microphone captures the audio data, which is then converted into text data in real time using the Google Cloud Speech-to-Text API, instantly recording what the user is saying.

[1216] 3. Emotion recognition

[1217] At the same time, the system uses the Microsoft Azure Cognitive Services Emotion API to analyze the user's emotions in real time from the tone and content of their voice. This API recognizes the speaker's emotions from voice data and can identify negative emotions such as anxiety or fear.

[1218] 4. Data transmission

[1219] The device encrypts the generated text data and emotion data using a secure communication protocol (e.g., HTTPS) and transmits it to the server, where it is protected from unauthorized access.

[1220] 5. Data Analysis

[1221] The server inputs the received text data into OpenAI's GPT-4 model to evaluate the likelihood of fraud or criminal activity, and analyzes the emotional data using Azure's Emotion API. This allows for highly accurate detection of fraud and criminal activity based on the user's emotional state and text data.

[1222] 6. Generating comprehensive evaluation results

[1223] The server integrates the analysis results of the generative AI model and the emotion evaluation engine to comprehensively evaluate the likelihood of fraud or criminal activity, and adjusts the importance of warnings and the priority of countermeasures based on the level of anxiety or fear expressed by the user as indicated by the emotion data.

[1224] 7. Providing warnings and suggested solutions

[1225] The server generates a warning message and suggested actions and sends it to the device, which then displays these messages to the user as a pop-up notification, such as "This may be a scam" or "Please check with an official contact."

[1226] 8. Record data and retrain

[1227] After the conversation is over, the device sends all data to the server, which stores it in a back-end database. This stored data will be used to retrain the generative AI model and emotion engine in the future. Periodic retraining ensures the system stays up to date with the latest fraud patterns and the user's emotional state.

[1228] Specific examples

[1229] Scenario: Fake billing call

[1230] Launching the app

[1231] The elderly user launches the dedicated app and prepares to receive a call. The device then starts the voice capture function.

[1232] Conversation initiation, transcription, and emotion recognition

[1233] The user receives a call and the conversation begins with the message, "Your payment is outstanding and you need to pay it immediately." The device captures the conversation in real time and instantly transcribes it into text using the Google Cloud Speech-to-Text API. At the same time, it uses Azure's Emotion API to detect anxiety from the user's tone and content of voice.

[1234] Data transmission and analysis

[1235] The device sends the generated text data and emotion data to a server, which then analyzes the received data using OpenAI's GPT-4 model and Azure Emotion API. The GPT-4 model detects fraudulent patterns such as "unpaid" and "pay immediately," while the emotion evaluation engine detects high levels of anxiety in the user.

[1236] Evaluation and warning message generation

[1237] The server determines that there is a high possibility of fraud and generates a warning message and a suggestion to "check with an official contact." Based on the emotional data, the importance of the warning is increased, urging even more caution.

[1238] Displaying warnings and suggested solutions

[1239] The terminal displays these messages to the user as pop-up notifications.

[1240] Recording Data and Retraining

[1241] After the conversation ends, the device sends all data to the server, which records it and uses it to retrain the generative AI model and emotion engine.

[1242] Prompt Sentence Examples

[1243] What should you do if you receive a call saying, "Please transfer the money right now"?

[1244] By inputting this prompt into a generative AI model, the system can suggest appropriate countermeasures.

[1245] The above is an embodiment of the present invention. This system effectively protects users from fraud and criminal activity and provides countermeasure suggestions according to the user's emotional state.

[1246] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1247] Step 1:

[1248] A user launches the dedicated app on a smartphone or tablet. The device displays the app's interface and prepares to start the voice capture and emotion recognition functions. The input is the user's operation, and the output is the display of the app's home screen, which includes a "Start phone call" button.

[1249] Step 2:

[1250] When a user receives a call, the device's microphone captures audio data. The input is the captured audio data, and the output is a real-time stream of that audio data. The device then sends the captured audio data to the Google Cloud Speech-to-Text API, which converts it into text data. Specifically, the audio data is analyzed in real time and instantly converted into text.

[1251] Step 3:

[1252] At the same time, the device uses the Emotion API from Microsoft Azure Cognitive Services to analyze the user's emotions in real time. The input is the captured voice data, and the output is the emotion recognition results. Specifically, the Emotion API identifies negative emotions from the tone and content of the voice.

[1253] Step 4:

[1254] The text data and emotion data generated by the device are encrypted using a secure communication protocol (e.g., HTTPS) and sent to the server. The input is text data and emotion data, and the output is the encrypted data. This ensures data security.

[1255] Step 5:

[1256] The server inputs the received text data into OpenAI's GPT-4 model to evaluate the likelihood of fraud or criminal activity. The input is the received text data, and the output is the analysis result. Specifically, the GPT-4 model detects keywords such as "unpaid" and "pay immediately" in the text.

[1257] Step 6:

[1258] The server analyzes the emotion data using an emotion evaluation engine. The input is the received emotion data, and the output is the analyzed emotion evaluation result. Specifically, the Emotion API evaluates the user's anxiety or fear level.

[1259] Step 7:

[1260] The server integrates the analysis results of the generative AI model and the emotion evaluation engine to generate a comprehensive evaluation result. The input is the analysis results of the text data and the evaluation results of the emotion data, and the output is a comprehensive evaluation report. Specifically, the server combines both sets of data to evaluate the likelihood and severity of fraud or criminal activity.

[1261] Step 8:

[1262] The server generates a warning message and suggested countermeasures based on the evaluation results and sends them to the terminal. The input is the overall evaluation result, and the output is a warning message and suggested countermeasures. Specifically, the server generates messages such as "This may be a scam" or "Please check with the official contact point."

[1263] Step 9:

[1264] The terminal displays the received warning message and suggested countermeasures to the user as a pop-up notification. The input is the warning message and suggested countermeasures sent from the server, and the output is the notification to the user. Specifically, the terminal displays a pop-up message on the screen to immediately notify the user of the warning.

[1265] Step 10:

[1266] After the conversation is over, the device sends all data to the server. The input is the generated text data and emotion data, and the output is data sent to the server. The server stores this data in a backend database. The stored data is used to retrain the generative AI model and emotion engine. This improves the accuracy of the system and allows it to adapt to the latest fraud patterns and the user's emotional state.

[1267] (Application example 2)

[1268] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1269] Fraud and criminal activity methods are evolving daily, resulting in a constant stream of fraud victims perpetrated over the phone or in face-to-face conversations. Elderly people and those with low digital literacy are particularly susceptible to scams, and are prone to financial loss and psychological damage. However, conventional fraud prevention systems struggle to provide real-time countermeasures and are unable to provide warnings or suggest countermeasures that take into account the user's emotional state. To solve this problem, a system is needed that converts speech into text in real time, uses a generative artificial intelligence model and an emotion engine to comprehensively evaluate the likelihood of fraud or criminal activity, and then provides appropriate warnings and suggests countermeasures to the user.

[1270] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting voice data into text data in real time, means for analyzing the text data using a generative artificial intelligence model and evaluating the possibility of fraud or criminal activity, and means for recognizing the user's emotional state using an emotion engine and displaying a warning message and suggested countermeasures to the user based on the evaluation results. This makes it possible to detect fraud or criminal activity in real time and issue appropriate warnings and suggest countermeasures according to the user's emotional state.

[1271] "Audio data" refers to audio signals uttered by a user recorded as digital information.

[1272] "Text data" is a collection of character information generated by analyzing voice data.

[1273] A "generative artificial intelligence model" is an algorithm that uses machine learning to analyze text data and understand and evaluate specific patterns and meanings.

[1274] The "emotion engine" is an algorithm that analyzes the user's emotional state from their voice and facial expressions and evaluates the results.

[1275] A "warning message" is an informational notice intended to inform the user of a high possibility of fraud or criminal activity.

[1276] "Measures Proposal" is an information notification that indicates specific actions or measures that the user should take based on the warning message.

[1277] A "server" is a computer system that analyzes voice data and text data and generates comprehensive evaluation results.

[1278] A "terminal" is a device used by a user, which has the function of capturing voice data and displaying warning messages and suggested countermeasures received from the server.

[1279] This invention is a system that converts voice data into text data in real time, evaluates the possibility of fraud or criminal activity using a generative AI model and an emotion engine, and presents warnings and countermeasure suggestions to users. This system is mainly composed of a server, a terminal, and a user.

[1280] System configuration

[1281] Terminal

[1282] The terminal is a device such as a smartphone or tablet that has a microphone for capturing audio data and a display for displaying text data and warning messages.

[1283] When a user initiates a phone or face-to-face conversation, the device captures voice data and converts it into text data in real time using a speech recognition engine.

[1284] server

[1285] The server is a central computer system that receives and analyzes the text data and emotion data sent from the terminals.

[1286] To analyze the text data, a generative AI model (such as BERT or GPT) is used. The generative AI model compares patterns of fraud and criminal activity with a database to assess whether the conversation is fraudulent. In addition, an emotion engine analyzes the user's emotional state based on the voice data sent from the device. This makes it possible to determine whether the user is feeling anxious or fearful.

[1287] User

[1288] A user is an individual who uses the system and is the sender of voice data. When a user answers a phone call or has a face-to-face conversation, the voice data is captured through the terminal and sent to the server. Analysis results and warning messages from the server are notified to the user through the terminal.

[1289] Program processing

[1290] 1. Capture and convert voice data: Capture voice using the device's microphone and convert it into text data in real time using the Google Speech Recognition API.

[1291] 2. Data Analysis: Send the text data and emotion data to the server, where the text data is analyzed using a generative AI model (e.g., BERT), and the emotion engine is used to analyze the user's emotional state.

[1292] 3. Evaluation and Notification: The server integrates the results of analysis by the generative AI model and emotion engine, and if there is a high possibility of fraud, a warning message is generated and sent to the device. The notification content may be something like "This is likely a scam" or "Please check with the official contact point."

[1293] Specific examples

[1294] Scenario: Fake billing call

[1295] 1. User receives a call: The elderly user receives a call and launches the app on their device.

[1296] 2. Real-time transcription of the conversation: The audio data of "It's unpaid and please pay it immediately" is captured and converted into text data in real time.

[1297] 3. Data transmission and analysis: Text data and emotion data are transmitted to the server, and the generative AI model detects fraud patterns such as "unpaid" and "pay immediately." At the same time, the emotion engine detects high levels of anxiety from the user.

[1298] 4. Integration of evaluation results and generation of warning: The server determines that there is a high possibility of fraud and generates a warning message such as "Please check with the official contact point."

[1299] 5. Warning Notifications: The device will display these messages to the user as a popup.

[1300] Prompt Sentence Examples

[1301] "Pay any outstanding amounts immediately."

[1302] Category labels: Scam, Safety

[1303] In this way, the system can provide users with real-time warnings and countermeasures against fraud and criminal activity, helping to keep them safe.

[1304] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1305] Step 1:

[1306] A user launches an app on a smartphone or tablet and starts a phone or face-to-face conversation. The device's microphone captures audio data. The input is the user's voice data, which is recorded as a digital audio signal by the app's microphone. The output is the captured voice data.

[1307] Step 2:

[1308] The device converts captured voice data into text data in real time. This conversion is performed using a speech recognition engine (e.g., Google Speech Recognition API). The input is the captured voice data, and the output is the converted text data.

[1309] Step 3:

[1310] The converted text data and emotion data detected from the user's voice tone and content are sent from the device to the server. The data is encrypted and sent using a secure communication protocol (e.g., SSL / TLS). The input is text data and emotion data, and the output is data that is sent securely.

[1311] Step 4:

[1312] The server analyzes the received text data using a generative AI model (e.g., BERT). This analysis evaluates the text data to detect patterns of fraud and criminal activity. The input is the submitted text data, and the output is the generative AI model's evaluation of the fraud and criminal activity.

[1313] Step 5:

[1314] In parallel, the server inputs the received emotion data into the emotion engine to analyze the user's emotional state. The emotion engine evaluates the user's negative emotions, such as anxiety, discomfort, and fear, from the voice data and text. The input is the transmitted emotion data, and the output is the analysis result of the user's emotional state.

[1315] Step 6:

[1316] The server integrates the evaluation results of the generative AI model and the analysis results of the emotion engine to comprehensively determine the likelihood of fraud or criminal activity. Depending on the evaluation results, a warning message and countermeasure proposals are generated. The input is the analysis results of text data and emotion data, and the output is the integrated evaluation results and the warning message and countermeasure proposals based on them.

[1317] Step 7:

[1318] The server generates a warning message and sends it to the terminal. The input is the warning message and the countermeasure proposal, and the output is the notification data that is securely transmitted.

[1319] Step 8:

[1320] The terminal displays a warning message and suggested countermeasures to the user as a pop-up message. This allows the user to receive warnings about possible fraud or criminal activity and appropriate countermeasures in real time. The input is the warning message and suggested countermeasures sent from the server, and the output is the pop-up message displayed on the terminal display.

[1321] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1322] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1323] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1324] [Fourth embodiment]

[1325] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1326] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1327] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1328] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1329] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1330] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1331] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1332] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1333] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1334] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1335] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1336] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1337] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1338] This invention is a system that converts voice data into text data in real time, evaluates the possibility of fraud or criminal activity using a generative artificial intelligence model, and presents a warning and countermeasure suggestions to the user. Specific embodiments will be described below.

[1339] System configuration and operation

[1340] 1. The user launches the app

[1341] Users launch a dedicated app on their smartphone or tablet.

[1342] The device will display the app interface and start the audio capture function.

[1343] When a user starts a phone or in-person conversation, the app automatically begins capturing audio.

[1344] 2. Real-time transcription of conversations

[1345] The device converts the captured audio into text data in real time, using voice recognition technology to instantly convert spoken words into written text.

[1346] Once the transcription is complete, the text data is sent to the server.

[1347] 3. Analysis and Evaluation of Text Data

[1348] The server inputs the received text data into a generative artificial intelligence model.

[1349] The generative AI model analyzes the content of the conversation by comparing it with a database of patterns of fraud and criminal activity.

[1350] If fraud or criminal activity is suspected, the server evaluates the results and generates a warning message.

[1351] 4. Present warnings and suggested solutions

[1352] The server returns data including a warning message and suggested countermeasures to the terminal.

[1353] The device will warn the user with a pop-up message and suggest appropriate measures, such as "This may be a scam" or "Please check with official contacts."

[1354] 5. Recording conversation patterns

[1355] The terminal sends the data of the completed conversation to the server, where it is recorded in the backend.

[1356] The server analyzes the stored data and uses it as data for retraining the generative AI model.

[1357] Specific examples

[1358] Scenario: Fake billing call

[1359] 1. Launch the app

[1360] An elderly user launches a dedicated app.

[1361] The user prepares to receive the call.

[1362] 2. Start a conversation and transcribe

[1363] The user receives a call and the conversation begins with the user being told that the outstanding amount is due and that they should pay it immediately.

[1364] The device captures the conversation in real time and instantly transcribes it into text.

[1365] 3. Sending and analyzing text data

[1366] The terminal transmits the generated text data to the server.

[1367] The data received by the server is analyzed using a generative AI model to detect fraudulent patterns such as "unpaid" and "pay immediately."

[1368] 4. Displaying warnings and suggested solutions

[1369] The server determines that there is a high possibility of fraud and generates a warning message and a suggestion to "check with the official contact point."

[1370] The terminal displays these messages to the user as popups.

[1371] 5. Recording conversation patterns

[1372] After the conversation is over, the device sends all data to the server.

[1373] The server records the data and uses it to retrain the generative AI model.

[1374] This system not only effectively protects users from fraud and criminal activity, but also allows the accuracy of the generative AI model to be constantly improved based on the collected data.The system covers both online and offline risks, making it effective for a wide range of users.

[1375] ---

[1376] The above is the explanation of the "Mode for Carrying Out the Invention" based on the scope of the claims. This content will help you understand the specific methods for carrying out the invention and you can use it as a reference to carry out the invention.

[1377] The processing flow will be explained below.

[1378] Step 1:

[1379] The user launches the dedicated app on their smartphone or tablet. The device displays the app interface and starts the audio capture function.

[1380] Step 2:

[1381] The user answers a phone call or starts a face-to-face conversation. The device begins capturing audio through the microphone. Once audio capture begins, the device transcribes the captured audio data in real time.

[1382] Step 3:

[1383] The device generates the transcribed text data and sends it to the server, where it is encrypted using a secure communication protocol.

[1384] Step 4:

[1385] The server inputs the received text data into a generative AI model, which then compares it with a database of fraud and criminal patterns and analyzes the conversation.

[1386] Step 5:

[1387] The generative AI model returns the analysis results to the server, which evaluates them and determines whether there is any potential for fraud or criminal activity.

[1388] Step 6:

[1389] The server generates a warning message based on the evaluation results, including specific warning content and suggested countermeasures for the user.

[1390] Step 7:

[1391] The server sends back data containing a warning message and suggested solutions to the terminal, which receives it and displays it to the user as a pop-up message.

[1392] Step 8:

[1393] The user ends the call. The device sends all conversation data (audio and transcription data) to the server.

[1394] Step 9:

[1395] The server records the data it receives and stores it in a back-end database, which can then be used to retrain the AI ​​model in the future.

[1396] Step 10:

[1397] The server periodically retrains the generative AI model based on new conversation patterns, thereby improving the model's detection accuracy.

[1398] ---

[1399] This is a detailed explanation of the program's processing steps, which will give you a clear understanding of how the system works and what specific processes are performed.

[1400] Example 1

[1401] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1402] Conventional methods have had difficulty detecting fraud and criminal activity in real time and quickly warning users. They also lacked the means to securely transmit voice data and properly return analysis results and warning messages to the device. Therefore, there was a need for an effective means for users to prevent themselves from becoming victims of fraud and criminal activity.

[1403] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1404] In this invention, the server includes means for converting voice data into text data in real time, means for analyzing the text data using a generative artificial intelligence model to evaluate the likelihood of fraud or criminal activity, means for displaying a warning message and suggested countermeasures to the user based on the evaluation results, means for transmitting text data transmitted from the user to the terminal to the server, means for analyzing the text data using the generative artificial intelligence model to detect the likelihood of fraud or criminal activity, means for returning the generated warning message and suggested countermeasures to the terminal, and means for recording the conversation data and using it as data for retraining, thereby enabling users to be effectively protected from fraud and criminal activity.

[1405] "Audio data" means data in digital form that contains human speech or other sound information.

[1406] "Text data" is data consisting of characters and symbols that is obtained as a result of converting voice data.

[1407] A "generative artificial intelligence model" is an artificial intelligence model with natural language processing capabilities that is trained based on a large dataset.

[1408] "Analysis" is the process performed to determine the possibility of fraud or criminal activity based on the input data.

[1409] A "warning message" is a message that is generated as a result of analysis to warn or alert the user.

[1410] "Measure suggestions" are specific actions or measures recommended to the user based on the warning message.

[1411] "User" refers to any individual or corporation that uses this system.

[1412] "Terminal" refers to an electronic device, such as a smartphone or tablet, that allows a user to access and use the system.

[1413] A "server" is a central computer system used to process voice and text data and provide information to users.

[1414] "Real-time" refers to processing that occurs almost simultaneously with the occurrence of an event.

[1415] "Conversation data" refers to data that includes captured speech and text data generated based on the speech.

[1416] "Retraining" is the process of retraining a generative artificial intelligence model using new data to improve its accuracy.

[1417] MODE FOR CARRYING OUT THE INVENTION

[1418] The present invention provides a system that converts voice data into text data in real time, evaluates the possibility of fraud or criminal activity using a generative artificial intelligence model, and presents a warning and countermeasure suggestions to the user. A specific embodiment of the present invention will be described below.

[1419] System configuration and operation

[1420] The user launches the app

[1421] Users launch a dedicated app on their smartphone or tablet. This app has a voice capture function, and when the user starts a phone or face-to-face conversation, the device begins capturing the voice in real time.

[1422] Voice capture to text conversion

[1423] The device converts the captured voice data into text data in real time using voice recognition software (e.g., Google Speech-to-Text API), and the converted text data is immediately sent to the server.

[1424] Text data analysis

[1425] The server inputs the received text data into a generative AI model (e.g., OpenAI GPT-4). The generative AI model analyzes the conversation by comparing it with a database of fraudulent and criminal patterns. If the analysis reveals a possibility of fraud or criminal activity, the server generates a warning message and suggests countermeasures.

[1426] Warnings and suggested solutions

[1427] The server generates a warning message and sends it back to the device, which displays it to the user as a pop-up message. Specific messages include "This may be a scam" and "Please check with your official contacts."

[1428] Recording and retraining conversation data

[1429] After the conversation is complete, the device sends all conversation data to the server, which records it in a database. This data is used to retrain the generative AI model, helping to improve the accuracy of the system.

[1430] Specific operation example

[1431] Scenario: Fake billing call

[1432] 1. Launch the app

[1433] An elderly user launches a dedicated app.

[1434] The user prepares to receive the call.

[1435] 2. Start a conversation and transcribe

[1436] The user receives a call and the conversation begins with the user being told that the outstanding amount is due and that they should pay it immediately.

[1437] The device captures the conversation in real time and instantly transcribes it into text.

[1438] 3. Sending and analyzing text data

[1439] The terminal transmits the generated text data to the server.

[1440] The data received by the server is analyzed using a generative artificial intelligence model to detect fraudulent patterns such as "non-payment" and "immediate payment."

[1441] 4. Displaying warnings and suggested solutions

[1442] The server determines that there is a high possibility of fraud and generates a warning message and a suggestion to "check with the official contact point."

[1443] The terminal displays these messages to the user as popups.

[1444] 5. Recording conversation patterns

[1445] After the conversation is over, the terminal sends all data to the server.

[1446] The server records the data and uses it to retrain the generative AI model.

[1447] In this way, the system of the present invention can consistently convert voice data in real time, analyze it using a generative AI model, and promptly warn users and suggest countermeasures. This system effectively protects users from fraud and criminal activity, and the accuracy of the system is continuously improved.

[1448] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1449] Step 1:

[1450] The user launches the app

[1451] Input: The user operates a smartphone or tablet device to launch a dedicated app.

[1452] What it does: The device displays the app interface and puts the audio capture feature into standby.

[1453] Output: The audio capture feature is enabled and audio is recorded when the user starts speaking.

[1454] Step 2:

[1455] The device captures audio in real time

[1456] Input: Ambient sounds, especially phone calls and face-to-face conversations you have.

[1457] What it does: Your device uses the app's audio capture feature to record audio data in real time.

[1458] Output: Recorded audio data.

[1459] Step 3:

[1460] The device converts the voice data into text data.

[1461] Input: Real-time recorded audio data.

[1462] How it works: Your device uses speech recognition software (e.g., Google Speech-to-Text API) to convert voice data into text.

[1463] Output: The converted text data.

[1464] Step 4:

[1465] The device sends text data to the server.

[1466] Input: The converted text data.

[1467] How it works: The device sends text data to the server using a secure communication protocol (e.g., HTTPS).

[1468] Output: Text data is sent to the server.

[1469] Step 5:

[1470] The server analyzes the text data using a generative AI model

[1471] Input: The text data sent to the server.

[1472] How it works: The server feeds text data into a generative artificial intelligence model (e.g., OpenAI GPT-4) to analyze it for potential fraud or criminal activity.

[1473] Output: Analysis results in assessment data of the likelihood of fraud or criminal activity.

[1474] Step 6:

[1475] The server evaluates the results and generates a warning message

[1476] Input: Analysis results generated by a generative AI model.

[1477] How it works: The server evaluates the analysis results and generates a warning message and suggested actions if there is a high possibility of fraud or criminal activity.

[1478] Output: Generated warning messages and suggested actions.

[1479] Step 7:

[1480] The server sends a warning message and suggested solutions back to the device.

[1481] Input: Generated warning message and suggested action.

[1482] Operation: The server returns a warning message and suggested solutions to the device as an HTTP response.

[1483] Output: A warning message and suggested solutions will be sent to your device.

[1484] Step 8:

[1485] The device displays a warning and suggests countermeasures to the user.

[1486] Input: The warning message and suggested action returned by the server.

[1487] How it works: The device displays a pop-up notification to the user with a warning message and suggested solutions.

[1488] Output: User confirms warning message and suggested action.

[1489] Step 9:

[1490] The device records conversation patterns and sends them to the server.

[1491] Input: Finished conversation data.

[1492] Operation: The terminal records the conversation data that has been completed and sends it to the server.

[1493] Output: Conversation data recorded on the server.

[1494] Step 10:

[1495] The server records the data and uses it to retrain the generative AI model.

[1496] Input: Conversation data sent to the server.

[1497] How it works: The server stores conversation data and uses it as retraining data for generative AI models.

[1498] Output: An updated generative AI model and conversation data recorded in a database.

[1499] (Application example 1)

[1500] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1501] In modern society, fraud and criminal acts are becoming increasingly sophisticated, and many people are falling victim to these methods. To address this situation, a system is needed that can detect fraud and criminal acts early and provide appropriate warnings and countermeasures. However, current technology does not provide a system that can analyze voice data in real time and immediately evaluate the possibility of fraud and provide a warning. Therefore, an objective of the present invention is to provide a system that can detect fraud and criminal acts in real time and provide a warning, allowing users to communicate safely.

[1502] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1503] In this invention, the server includes means for converting voice data into text data in real time, means for analyzing the text data using a generative artificial intelligence model to evaluate the possibility of fraud or criminal activity, means for displaying a warning message and countermeasure suggestions to the user based on the evaluation results, means for providing a voice capture function, means for converting voice into text using voice recognition technology, means for analyzing the text by matching it with fraud patterns, and means for sending a warning message to the user using communication means, thereby enabling the user to be aware of the possibility of fraud or criminal activity in real time and take appropriate countermeasures.

[1504] "Audio data" refers to continuous sound signals obtained from conversations, communications, etc.

[1505] "Text data" is voice data converted into a string of characters in a language.

[1506] A "generative artificial intelligence model" is an AI technology that learns large amounts of data and generates and analyzes sentences and information.

[1507] "Voice recognition technology" is a technology that converts voice data into text in real time.

[1508] "Fraud patterns" are data that indicate combinations of words and phrases associated with fraud.

[1509] A "warning message" is a notification that alerts the user to danger or problems.

[1510] "Measure suggestions" are advice that shows specific actions or methods that users should take.

[1511] "User" refers to any person or organization that uses this system.

[1512] The "audio capture function" is a function for collecting audio in real time.

[1513] "Communication means" refers to the technology and protocols used to exchange information.

[1514] A "server" is a computer system that processes and stores data and provides services over a network.

[1515] "Terminal" refers to a device or equipment that is directly operated by a user.

[1516] "Analysis results" refers to the results of analysis using a generative artificial intelligence model.

[1517] MODE FOR CARRYING OUT THE INVENTION

[1518] This invention is a security system for users that converts voice data into text data in real time, analyzes the text data using a generative artificial intelligence model to evaluate the possibility of fraud or criminal activity, and provides the user with a warning and suggestions for countermeasures. This system is mainly composed of the following elements.

[1519] 1. Hardware and Software Selection

[1520] The following hardware and software are used to realize this system.

[1521] Smartphone terminal: A device operated by a user.

[1522] Microphone: Used to capture audio data.

[1523] Server: Processes data and manages the database.

[1524] Google Cloud Speech-to-Text API: Provides speech recognition technology.

[1525] Generative AI models (e.g., OpenAI GPT): Analyze text data to assess the likelihood of fraud or criminal activity.

[1526] Twilio API: A communication method for sending alert messages to users.

[1527] 2. Explanation of program processing

[1528] The main processing flow of the program of this system is explained below.

[1529] User voice input

[1530] The user starts a conversation using a smartphone. The device's voice capture function is activated, capturing the conversation's audio data in real time. This allows the user's conversation to be input into the system in real time and prepared for analysis.

[1531] Converting audio data to text

[1532] The server converts the captured audio data into text data using the Google Cloud Speech-to-Text API, which then sends the audio data to the server as text data.

[1533] Text data analysis

[1534] The server uses a generative AI model (OpenAI GPT) to analyze the converted text data. This identifies patterns of fraud and criminal activity and assesses risk. This analysis process is performed by using specific keywords and phrases as prompts. The following are example prompts:

[1535] Rate the following text as a scam: Your credit card details may have been stolen. Check now

[1536] Warning messages and suggested solutions

[1537] If the server determines based on the analysis results that there is a high possibility of fraud or criminal activity, it uses the Twilio API to send a warning message to the user's device. The warning message also includes specific suggestions for countermeasures. For example, a message such as "This may be a scam. Please contact an official contact point" may be displayed.

[1538] Adding specific examples

[1539] As a concrete example, let's consider a scenario in which an elderly user receives a fraudulent phone call. When the user receives a call stating, "You have an outstanding payment and need to pay it immediately," the audio is captured in real time and converted to text using the Google Cloud Speech-to-Text API. The OpenAI GPT model then evaluates the likelihood of fraud and, if it determines there is a high risk, sends a warning message to the user's smartphone via the Twilio API stating, "There is a high possibility of fraud. Please check with your official contact." In this way, the user can quickly respond to the risk of fraud.

[1540] This system will enable users to receive real-time warnings about fraud and criminal activity, which is expected to help prevent people from falling victim to fraud.

[1541] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1542] Step 1:

[1543] A user launches an app on their smartphone and starts the voice capture function. When the user starts a phone call or face-to-face conversation, the device starts capturing voice data in real time. The input is the user's voice data, and the output is the captured voice data.

[1544] Step 2:

[1545] The captured voice data is sent to the Google Cloud Speech-to-Text API, where it is converted into text data using speech recognition technology. The input is voice data, and the output is converted text data. This conversion process analyzes the linguistic characteristics of the voice data and the speaker's voice and converts it into text.

[1546] Step 3:

[1547] The converted text data is sent to a server and analyzed by a generative AI model (OpenAI GPT). The input is text data and the output is the analysis result. In this step, the text data is matched with fraud patterns to evaluate the likelihood of fraud or criminal activity. Specifically, the model is queried using a specific prompt sentence.

[1548] Rate the following text as a scam: Your credit card details may have been stolen. Check now

[1549] Step 4:

[1550] The server generates a warning message and countermeasure suggestions based on the analysis results. The input is the analysis results, and the output is a warning message and countermeasure suggestions. If the analysis results indicate a possibility of fraud, a warning message is created and saved as data to be sent to the user.

[1551] Step 5:

[1552] The generated warning message and suggested measures are sent to the user's smartphone using the Twilio API. The input is the warning message and suggested measures, and the output is the warning message displayed on the user's smartphone. Specifically, an SMS is sent via the Twilio API, and a pop-up message is displayed on the user's device.

[1553] Step 6:

[1554] After the conversation is over, all collected data is sent to the server and stored in a database. The input is the conversation data and analysis results, and the output is the data stored in the database. The stored data is used to retrain the generative AI model, contributing to improving the accuracy of the system.

[1555] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1556] This invention is a system that converts voice data into text data in real time, evaluates the possibility of fraud or criminal activity using a generative artificial intelligence model and an emotion engine, and presents a warning and countermeasure suggestions to the user. Specific embodiments are described below.

[1557] System configuration and operation

[1558] 1. The user launches the app

[1559] Users launch a dedicated app on their smartphone or tablet.

[1560] The device will display the app interface and begin voice capture and emotion recognition functions.

[1561] 2. Real-time conversation transcription and emotion recognition

[1562] The user answers a phone call or starts a face-to-face conversation. The device begins capturing audio through the microphone.

[1563] The device converts voice data into text data in real time, while the emotion engine simultaneously recognizes emotions from the user's voice and facial expressions.

[1564] 3. Sending text data and emotion data

[1565] The device then transmits the generated text data and emotion data to a server, where the data is encrypted using a secure communication protocol.

[1566] 4. Analysis of text data and emotion data

[1567] The server inputs the received text data into a generative AI model, while inputting the emotion data into an emotion evaluation engine.

[1568] The generative AI model analyzes the conversation content against a database of fraud and criminal patterns, while an emotion assessment engine simultaneously analyzes the user's emotional state.

[1569] 5. Generating comprehensive evaluation results

[1570] The server integrates the analysis results of the generative AI model and the emotion evaluation engine to comprehensively assess the likelihood of fraud or criminal activity.

[1571] If the emotional data indicates strong negative emotions such as anxiety or fear in the user, the importance of the warning and the priority of the countermeasures will be adjusted.

[1572] 6. Providing warnings and suggested solutions

[1573] The server generates a warning message and a countermeasure proposal and sends them to the terminal.

[1574] The device will display a warning to the user as a pop-up message and suggest appropriate measures, such as "This may be a scam" or "Please check with official contacts."

[1575] 7. Recording speech patterns and emotional data

[1576] When the conversation is finished, the terminal sends all data to the server.

[1577] The server records the received data and stores it in a back-end database, which can be used to retrain future AI models and emotion engines.

[1578] 8. Retraining AI models and emotion engines

[1579] The server periodically retrains the generative AI model and emotion engine with new speech patterns and emotion data, thereby improving the accuracy of the model and engine.

[1580] Specific examples

[1581] Scenario: Fake billing call

[1582] 1. Launch the app

[1583] An elderly user launches a dedicated app.

[1584] The user prepares to receive the call.

[1585] 2. Conversation initiation, transcription, and emotion recognition

[1586] The user receives a call and the conversation begins with the user being told that the outstanding amount is due and that they should pay it immediately.

[1587] The device captures conversations in real time and instantly transcribes them into text, while simultaneously detecting anxiety from the user's tone and content of voice.

[1588] 3. Data submission and analysis

[1589] The terminal transmits the generated text data and emotion data to the server.

[1590] The data received by the server is analyzed using a generative AI model and emotion evaluation engine.

[1591] A generative AI model detects fraud patterns such as "non-payment" and "pay now," while a sentiment assessment engine detects high levels of anxiety in users.

[1592] 4. Comprehensive evaluation and warning message generation

[1593] The server determines that there is a high possibility of fraud and generates a warning message and a suggestion to "check with an official contact." Based on the emotional data, the importance of the warning is increased, urging even more caution.

[1594] 5. Displaying warnings and suggested solutions

[1595] The terminal displays these messages to the user as popups.

[1596] 6. Record data and retrain

[1597] After the conversation ends, the device sends all data to the server.

[1598] The server records the data and uses it to retrain the generative AI model and emotion engine.

[1599] This system not only effectively protects users from fraud and criminal activity, but also suggests countermeasures based on the user's emotional state. This increases users' sense of security and helps prevent fraud from occurring. Furthermore, the accuracy of the system can be constantly improved based on the collected data.

[1600] ---

[1601] The above is an explanation of the "Mode for Carrying Out the Invention" based on the scope of the patent claims for the invention that combines an emotion engine. This content will help you understand the specific implementation methods of the invention and you can use it as a reference to carry out the invention.

[1602] The processing flow will be explained below.

[1603] Step 1:

[1604] The user launches the dedicated app on their smartphone or tablet, which displays the app's interface and starts the voice capture and emotion recognition functions.

[1605] Step 2:

[1606] When a user answers a phone call or starts a face-to-face conversation, the device begins capturing audio through the microphone. Audio capture begins immediately and the captured audio data is converted into text data in real time. At the same time, the emotion engine recognizes emotions from the user's voice and facial expressions in real time.

[1607] Step 3:

[1608] The device then transmits the generated text data and emotion data to a server, where the data is encrypted using a secure communication protocol.

[1609] Step 4:

[1610] The server inputs the received text data into a generative AI model and begins analysis. Meanwhile, it inputs the emotion data into an emotion evaluation engine to evaluate the user's emotional state.

[1611] Step 5:

[1612] The generative AI model analyzes the text data by comparing it with a database of fraud and criminal patterns, while the emotion assessment engine evaluates the user's emotional state, particularly anxiety and fear, and integrates this data into the generative AI model's analysis.

[1613] Step 6:

[1614] The server combines the analysis results of the generated AI model and the emotion evaluation engine to comprehensively evaluate the likelihood of fraud or criminal activity, adjust the importance and content of warning messages based on the emotion data, and generate the final evaluation result.

[1615] Step 7:

[1616] The server generates a warning message and suggested countermeasures and sends them to the device. The device receives the message and displays a pop-up warning to the user, suggesting appropriate countermeasures, such as "This may be a scam" or "Please check with the official contacts."

[1617] Step 8:

[1618] When the user ends the call, the device transmits all conversation data (voice and transcription data) and emotion data to the server, where the data is again encrypted and transmitted via a secure communication protocol.

[1619] Step 9:

[1620] The server records the received data and stores it in a back-end database, which then uses it to retrain the generative AI model and emotion engine.

[1621] Step 10:

[1622] The server periodically retrains the generative AI model and emotion engine based on new speech patterns and emotion data, thereby improving the accuracy of the model and engine.

[1623] ---

[1624] The above is a concrete explanation of the processing steps of the invention that combines the emotion engine. This allows you to clearly understand how the system operates and the specific processing that is performed.

[1625] Example 2

[1626] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1627] Conventional systems could convert voice data into text data in real time to assess the likelihood of fraud or criminal activity, but they were unable to adjust the severity of warnings or the priority of countermeasures while taking the user's emotional state into account. As a result, it was not possible to propose appropriate countermeasures based on the user's emotional state, making it difficult to sufficiently reduce the risk of fraud or criminal activity. The purpose of this invention is to provide an effective fraud and criminal activity assessment system that also takes the user's emotional state into account.

[1628] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for analyzing voice data and text data and recognizing the user's emotional state in real time, a means for analyzing the text data using a generative artificial intelligence model and evaluating the possibility of fraud or criminal activity, and a means for integrating the analysis results and the user's emotional state to adjust the importance of the warning and the priority of countermeasures. This makes it possible to more accurately evaluate the possibility of fraud or criminal activity and to provide appropriate warnings and propose countermeasures according to the user's emotional state.

[1629] "Voice data" refers to digital audio information captured from a user's speech via a microphone.

[1630] "Text data" is digital document information that is created by converting voice data into text in real time.

[1631] A "generative artificial intelligence model" is a machine learning algorithm used to analyze text data and understand specific patterns and meanings.

[1632] "Emotional state" refers to the psychological and emotional state of the user, which is analyzed from the user's tone of voice, facial expression, etc.

[1633] "Real-time" refers to operations and processing occurring instantly with minimal delay.

[1634] "Fraud and criminal activity" refers to any activity that involves using dishonest means to deceive others for profit or that violates the law.

[1635] "Evaluation means" refers to techniques or methods for analyzing specific information based on input data and deriving results.

[1636] A "warning message" is a notification message that alerts the user and notifies them of the existence of a risk.

[1637] A "measure proposal" is a proposal that instructs the user on specific actions and methods to take in response to a detected risk.

[1638] "Encryption means" is a technology that encrypts data using a certain algorithm to transmit the data securely, making the contents of the data unintelligible to third parties.

[1639] A "server" is a computer system for storing, analyzing, and providing data.

[1640] A "terminal" is a device (e.g., a smartphone or tablet) that a user directly operates and that communicates with a server.

[1641] A "secure communication protocol" is a standardized procedure or rule for securely communicating data (e.g., HTTPS).

[1642] This invention is a system that converts voice data into text data in real time, uses a generative artificial intelligence model and an emotion evaluation engine to evaluate the possibility of fraud or criminal activity, and presents warnings and suggested countermeasures to the user.

[1643] System configuration and operation

[1644] 1. Launch the app

[1645] The user launches the dedicated app on their smartphone or tablet. The device displays the app's interface and is ready to begin voice capture and emotion recognition. The app has a user-friendly interface, including a button that says "Start Call."

[1646] 2. Audio data capture and conversion

[1647] When a user answers a call, the device's microphone captures the audio data, which is then converted into text data in real time using the Google Cloud Speech-to-Text API, instantly recording what the user is saying.

[1648] 3. Emotion recognition

[1649] At the same time, the system uses the Microsoft Azure Cognitive Services Emotion API to analyze the user's emotions in real time from the tone and content of their voice. This API recognizes the speaker's emotions from voice data and can identify negative emotions such as anxiety or fear.

[1650] 4. Data transmission

[1651] The device encrypts the generated text data and emotion data using a secure communication protocol (e.g., HTTPS) and transmits it to the server, where it is protected from unauthorized access.

[1652] 5. Data Analysis

[1653] The server inputs the received text data into OpenAI's GPT-4 model to evaluate the likelihood of fraud or criminal activity, and analyzes the emotional data using Azure's Emotion API. This allows for highly accurate detection of fraud and criminal activity based on the user's emotional state and text data.

[1654] 6. Generating comprehensive evaluation results

[1655] The server integrates the analysis results of the generative AI model and the emotion evaluation engine to comprehensively evaluate the likelihood of fraud or criminal activity, and adjusts the importance of warnings and the priority of countermeasures based on the level of anxiety or fear expressed by the user as indicated by the emotion data.

[1656] 7. Providing warnings and suggested solutions

[1657] The server generates a warning message and suggested actions and sends it to the device, which then displays these messages to the user as a pop-up notification, such as "This may be a scam" or "Please check with an official contact."

[1658] 8. Record data and retrain

[1659] After the conversation is over, the device sends all data to the server, which stores it in a back-end database. This stored data will be used to retrain the generative AI model and emotion engine in the future. Periodic retraining ensures the system stays up to date with the latest fraud patterns and the user's emotional state.

[1660] Specific examples

[1661] Scenario: Fake billing call

[1662] Launching the app

[1663] The elderly user launches the dedicated app and prepares to receive a call. The device then starts the voice capture function.

[1664] Conversation initiation, transcription, and emotion recognition

[1665] The user receives a call and the conversation begins with the message, "Your payment is outstanding and you need to pay it immediately." The device captures the conversation in real time and instantly transcribes it into text using the Google Cloud Speech-to-Text API. At the same time, it uses Azure's Emotion API to detect anxiety from the user's tone and content of voice.

[1666] Data transmission and analysis

[1667] The device sends the generated text data and emotion data to a server, which then analyzes the received data using OpenAI's GPT-4 model and Azure Emotion API. The GPT-4 model detects fraudulent patterns such as "unpaid" and "pay immediately," while the emotion evaluation engine detects high levels of anxiety in the user.

[1668] Evaluation and warning message generation

[1669] The server determines that there is a high possibility of fraud and generates a warning message and a suggestion to "check with an official contact." Based on the emotional data, the importance of the warning is increased, urging even more caution.

[1670] Displaying warnings and suggested solutions

[1671] The terminal displays these messages to the user as pop-up notifications.

[1672] Recording Data and Retraining

[1673] After the conversation ends, the device sends all data to the server, which records it and uses it to retrain the generative AI model and emotion engine.

[1674] Prompt Sentence Examples

[1675] What should you do if you receive a call saying, "Please transfer the money right now"?

[1676] By inputting this prompt into a generative AI model, the system can suggest appropriate countermeasures.

[1677] The above is an embodiment of the present invention. This system effectively protects users from fraud and criminal activity and provides countermeasure suggestions according to the user's emotional state.

[1678] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1679] Step 1:

[1680] A user launches the dedicated app on a smartphone or tablet. The device displays the app's interface and prepares to start the voice capture and emotion recognition functions. The input is the user's operation, and the output is the display of the app's home screen, which includes a "Start phone call" button.

[1681] Step 2:

[1682] When a user receives a call, the device's microphone captures audio data. The input is the captured audio data, and the output is a real-time stream of that audio data. The device then sends the captured audio data to the Google Cloud Speech-to-Text API, which converts it into text data. Specifically, the audio data is analyzed in real time and instantly converted into text.

[1683] Step 3:

[1684] At the same time, the device uses the Emotion API from Microsoft Azure Cognitive Services to analyze the user's emotions in real time. The input is the captured voice data, and the output is the emotion recognition results. Specifically, the Emotion API identifies negative emotions from the tone and content of the voice.

[1685] Step 4:

[1686] The text data and emotion data generated by the device are encrypted using a secure communication protocol (e.g., HTTPS) and sent to the server. The input is text data and emotion data, and the output is the encrypted data. This ensures data security.

[1687] Step 5:

[1688] The server inputs the received text data into OpenAI's GPT-4 model to evaluate the likelihood of fraud or criminal activity. The input is the received text data, and the output is the analysis result. Specifically, the GPT-4 model detects keywords such as "unpaid" and "pay immediately" in the text.

[1689] Step 6:

[1690] The server analyzes the emotion data using an emotion evaluation engine. The input is the received emotion data, and the output is the analyzed emotion evaluation result. Specifically, the Emotion API evaluates the user's anxiety or fear level.

[1691] Step 7:

[1692] The server integrates the analysis results of the generative AI model and the emotion evaluation engine to generate a comprehensive evaluation result. The input is the analysis results of the text data and the evaluation results of the emotion data, and the output is a comprehensive evaluation report. Specifically, the server combines both sets of data to evaluate the likelihood and severity of fraud or criminal activity.

[1693] Step 8:

[1694] The server generates a warning message and suggested countermeasures based on the evaluation results and sends them to the terminal. The input is the overall evaluation result, and the output is a warning message and suggested countermeasures. Specifically, the server generates messages such as "This may be a scam" or "Please check with the official contact point."

[1695] Step 9:

[1696] The terminal displays the received warning message and suggested countermeasures to the user as a pop-up notification. The input is the warning message and suggested countermeasures sent from the server, and the output is the notification to the user. Specifically, the terminal displays a pop-up message on the screen to immediately notify the user of the warning.

[1697] Step 10:

[1698] After the conversation is over, the device sends all data to the server. The input is the generated text data and emotion data, and the output is data sent to the server. The server stores this data in a backend database. The stored data is used to retrain the generative AI model and emotion engine. This improves the accuracy of the system and allows it to adapt to the latest fraud patterns and the user's emotional state.

[1699] (Application example 2)

[1700] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1701] Fraud and criminal activity methods are evolving daily, resulting in a constant stream of fraud victims perpetrated over the phone or in face-to-face conversations. Elderly people and those with low digital literacy are particularly susceptible to scams, and are prone to financial loss and psychological damage. However, conventional fraud prevention systems struggle to provide real-time countermeasures and are unable to provide warnings or suggest countermeasures that take into account the user's emotional state. To solve this problem, a system is needed that converts speech into text in real time, uses a generative artificial intelligence model and an emotion engine to comprehensively evaluate the likelihood of fraud or criminal activity, and then provides appropriate warnings and suggests countermeasures to the user.

[1702] The identification process by the identification processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for converting voice data into text data in real time, means for analyzing the text data using a generative artificial intelligence model and evaluating the possibility of fraud or criminal activity, and means for recognizing the user's emotional state using an emotion engine and displaying a warning message and suggested countermeasures to the user based on the evaluation results. This makes it possible to detect fraud or criminal activity in real time and issue appropriate warnings and suggest countermeasures according to the user's emotional state.

[1703] "Audio data" refers to audio signals uttered by a user recorded as digital information.

[1704] "Text data" is a collection of character information generated by analyzing voice data.

[1705] A "generative artificial intelligence model" is an algorithm that uses machine learning to analyze text data and understand and evaluate specific patterns and meanings.

[1706] The "emotion engine" is an algorithm that analyzes the user's emotional state from their voice and facial expressions and evaluates the results.

[1707] A "warning message" is an informational notice intended to inform the user of a high possibility of fraud or criminal activity.

[1708] "Measures Proposal" is an information notification that indicates specific actions or measures that the user should take based on the warning message.

[1709] A "server" is a computer system that analyzes voice data and text data and generates comprehensive evaluation results.

[1710] A "terminal" is a device used by a user, which has the function of capturing voice data and displaying warning messages and suggested countermeasures received from the server.

[1711] This invention is a system that converts voice data into text data in real time, evaluates the possibility of fraud or criminal activity using a generative AI model and an emotion engine, and presents warnings and countermeasure suggestions to users. This system is mainly composed of a server, a terminal, and a user.

[1712] System configuration

[1713] Terminal

[1714] The terminal is a device such as a smartphone or tablet that has a microphone for capturing audio data and a display for displaying text data and warning messages.

[1715] When a user initiates a phone or face-to-face conversation, the device captures voice data and converts it into text data in real time using a speech recognition engine.

[1716] server

[1717] The server is a central computer system that receives and analyzes the text data and emotion data sent from the terminals.

[1718] To analyze the text data, a generative AI model (such as BERT or GPT) is used. The generative AI model compares patterns of fraud and criminal activity with a database to assess whether the conversation is fraudulent. In addition, an emotion engine analyzes the user's emotional state based on the voice data sent from the device. This makes it possible to determine whether the user is feeling anxious or fearful.

[1719] User

[1720] A user is an individual who uses the system and is the sender of voice data. When a user answers a phone call or has a face-to-face conversation, the voice data is captured through the terminal and sent to the server. Analysis results and warning messages from the server are notified to the user through the terminal.

[1721] Program processing

[1722] 1. Capture and convert voice data: Capture voice using the device's microphone and convert it into text data in real time using the Google Speech Recognition API.

[1723] 2. Data Analysis: Send the text data and emotion data to the server, where the text data is analyzed using a generative AI model (e.g., BERT), and the emotion engine is used to analyze the user's emotional state.

[1724] 3. Evaluation and Notification: The server integrates the results of analysis by the generative AI model and emotion engine, and if there is a high possibility of fraud, a warning message is generated and sent to the device. The notification content may be something like "This is likely a scam" or "Please check with the official contact point."

[1725] Specific examples

[1726] Scenario: Fake billing call

[1727] 1. User receives a call: The elderly user receives a call and launches the app on their device.

[1728] 2. Real-time transcription of the conversation: The audio data of "It's unpaid and please pay it immediately" is captured and converted into text data in real time.

[1729] 3. Data transmission and analysis: Text data and emotion data are transmitted to the server, and the generative AI model detects fraud patterns such as "unpaid" and "pay immediately." At the same time, the emotion engine detects high levels of anxiety from the user.

[1730] 4. Integration of evaluation results and generation of warning: The server determines that there is a high possibility of fraud and generates a warning message such as "Please check with the official contact point."

[1731] 5. Warning Notifications: The device will display these messages to the user as a popup.

[1732] Prompt Sentence Examples

[1733] "Pay any outstanding amounts immediately."

[1734] Category labels: Scam, Safety

[1735] In this way, the system can provide users with real-time warnings and countermeasures against fraud and criminal activity, helping to keep them safe.

[1736] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1737] Step 1:

[1738] A user launches an app on a smartphone or tablet and starts a phone or face-to-face conversation. The device's microphone captures audio data. The input is the user's voice data, which is recorded as a digital audio signal by the app's microphone. The output is the captured voice data.

[1739] Step 2:

[1740] The device converts captured voice data into text data in real time. This conversion is performed using a speech recognition engine (e.g., Google Speech Recognition API). The input is the captured voice data, and the output is the converted text data.

[1741] Step 3:

[1742] The converted text data and emotion data detected from the user's voice tone and content are sent from the device to the server. The data is encrypted and sent using a secure communication protocol (e.g., SSL / TLS). The input is text data and emotion data, and the output is data that is sent securely.

[1743] Step 4:

[1744] The server analyzes the received text data using a generative AI model (e.g., BERT). This analysis evaluates the text data to detect patterns of fraud and criminal activity. The input is the submitted text data, and the output is the generative AI model's evaluation of the fraud and criminal activity.

[1745] Step 5:

[1746] In parallel, the server inputs the received emotion data into the emotion engine to analyze the user's emotional state. The emotion engine evaluates the user's negative emotions, such as anxiety, discomfort, and fear, from the voice data and text. The input is the transmitted emotion data, and the output is the analysis result of the user's emotional state.

[1747] Step 6:

[1748] The server integrates the evaluation results of the generative AI model and the analysis results of the emotion engine to comprehensively determine the likelihood of fraud or criminal activity. Depending on the evaluation results, a warning message and countermeasure proposals are generated. The input is the analysis results of text data and emotion data, and the output is the integrated evaluation results and the warning message and countermeasure proposals based on them.

[1749] Step 7:

[1750] The server generates a warning message and sends it to the terminal. The input is the warning message and the countermeasure proposal, and the output is the notification data that is securely transmitted.

[1751] Step 8:

[1752] The terminal displays a warning message and suggested countermeasures to the user as a pop-up message. This allows the user to receive warnings about possible fraud or criminal activity and appropriate countermeasures in real time. The input is the warning message and suggested countermeasures sent from the server, and the output is the pop-up message displayed on the terminal display.

[1753] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1754] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1755] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1756] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1757] FIG. 9 illustrates an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and behaviors arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1758] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1759] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1760] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1761] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1762] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1763] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1764] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1765] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1766] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1767] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1768] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1769] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1770] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1771] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1772] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1773] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1774] The following is further disclosed regarding the above embodiment.

[1775] (Claim 1)

[1776] A means for converting voice data into text data in real time;

[1777] a means for analyzing text data using a generative artificial intelligence model to assess the likelihood of fraud or criminal activity;

[1778] means for displaying a warning message and suggested measures to a user based on the evaluation result;

[1779] A system including:

[1780] (Claim 2)

[1781] 2. The system of claim 1, further comprising encryption means for securely communicating audio data.

[1782] (Claim 3)

[1783] 2. The system according to claim 1, further comprising means for transmitting analysis results and warning messages from the server to the terminal.

[1784] (Claim 4)

[1785] 10. The system of claim 1, further comprising means for periodically retraining the generative artificial intelligence model with new data.

[1786] (Claim 5)

[1787] 10. The system of claim 1, further comprising means for matching said text data with a database of patterns of fraud and criminal activity.

[1788] "Example 1"

[1789] (Claim 1)

[1790] A means for converting voice data into text data in real time;

[1791] a means for analyzing text data using a generative artificial intelligence model to assess the likelihood of fraud or criminal activity;

[1792] means for displaying a warning message and suggested measures to a user based on the evaluation result;

[1793] means for transmitting text data transmitted from a user to a terminal to a server;

[1794] A means of analyzing text data using a generative artificial intelligence model to detect potential fraud or criminal activity;

[1795] means for returning the generated warning message and countermeasure proposal to the terminal;

[1796] a means for recording conversational data and using it as retraining data;

[1797] A system including:

[1798] (Claim 2)

[1799] 10. The system of claim 1, including encryption means for securely communicating the voice data.

[1800] (Claim 3)

[1801] 10. The system of claim 1, further comprising means for transmitting analysis results and warning messages from the server to the terminal.

[1802] "Application Example 1"

[1803] (Claim 1)

[1804] A means for converting voice data into text data in real time;

[1805] a means for analyzing text data using a generative artificial intelligence model to assess the likelihood of fraud or criminal activity;

[1806] means for displaying a warning message and suggested measures to a user based on the evaluation result;

[1807] a means for providing audio capture functionality;

[1808] a means for converting speech to text using speech recognition technology;

[1809] a means for analyzing the text against fraud patterns;

[1810] means for sending a warning message to a user using a communication means;

[1811] A system including:

[1812] (Claim 2)

[1813] 2. The system of claim 1, further comprising encryption means for securely communicating audio data.

[1814] (Claim 3)

[1815] 2. The system according to claim 1, further comprising means for transmitting analysis results and warning messages from the server to the terminal.

[1816] "Example 2: Combining Emotion Engines"

[1817] (Claim 1)

[1818] A means for converting voice data into text data in real time;

[1819] means for analyzing voice data and text data and recognizing the emotional state of a user in real time;

[1820] a means for analyzing text data using a generative artificial intelligence model to assess the likelihood of fraud or criminal activity;

[1821] means for displaying a warning message and suggested measures to a user based on the evaluation result;

[1822] A means for integrating the analysis results and the user's emotional state to adjust the importance of warnings and the priority of countermeasures;

[1823] A system including:

[1824] (Claim 2)

[1825] 10. The system of claim 1, further comprising encryption means for securely communicating voice and text data.

[1826] (Claim 3)

[1827] 2. The system according to claim 1, further comprising means for transmitting analysis results and warning messages from the server to the terminal.

[1828] "Application example 2 when combining emotion engines"

[1829] (Claim 1)

[1830] A means for converting voice data into text data in real time;

[1831] a means for analyzing text data using a generative artificial intelligence model to assess the likelihood of fraud or criminal activity;

[1832] means for recognizing the emotional state of a user using an emotion engine and displaying a warning message and suggested measures to the user based on the evaluation result;

[1833] means for securely transmitting the voice data and analysis results to a server and transmitting a comprehensive evaluation result from the server to the terminal;

[1834] A system including:

[1835] (Claim 2)

[1836] 2. The system according to claim 1, further comprising means for transmitting the analysis results and warning messages from the server to the terminal.

[1837] (Claim 3)

[1838] 2. The system of claim 1, wherein data is communicated securely using encryption means. [Explanation of symbols]

[1839] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A means for converting voice data into text data in real time; a means for analyzing text data using a generative artificial intelligence model to assess the likelihood of fraud or criminal activity; means for displaying a warning message and suggested measures to a user based on the evaluation result; A system including:

2. 2. The system of claim 1, further comprising encryption means for securely communicating audio data.

3. 2. The system according to claim 1, further comprising means for transmitting analysis results and warning messages from the server to the terminal.

4. 10. The system of claim 1, further comprising means for periodically retraining the generative artificial intelligence model with new data.

5. 10. The system of claim 1, further comprising means for matching said text data with a database of patterns of fraud and criminal activity.

Images