System

Abstract

A system is provided.SOLUTION: A system comprising: means for receiving voice data transmitted from a terminal in real time and converting the voice data into text data using a voice recognition engine; means for analyzing the converted text data using a natural language processing algorithm and detecting suspicious patterns; means for sending visual and audio alerts to the user's terminal if suspicious patterns are detected; and means for sending notifications to pre-registered family and guardian terminals if suspicious patterns are detected.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] The present invention relates to a special fraud prevention system targeted at the elderly. In modern society, telephone fraud is on the rise, with many of the victims being elderly. It is difficult for elderly people to calmly deal with these fraudulent acts, so quick and effective prevention measures are needed. In order to protect the elderly from such fraudulent acts, the present invention provides a system that monitors call content in real time, detects suspicious patterns, and issues warnings. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by the following means. First, the system includes a means for receiving voice data transmitted from a terminal in real time and converting the voice data into text data using a voice recognition engine. Next, the system includes a means for analyzing the converted text data using a natural language processing algorithm to detect suspicious patterns. The system further includes a means for sending visual and audio alerts to the user's terminal when a suspicious pattern is detected, and a means for sending notifications to the terminals of pre-registered family members or guardians when a suspicious pattern is detected. This allows elderly people to receive prompt warnings and take action before they fall victim to fraud. Furthermore, the system provides an interface that allows users to add, delete, and edit phone numbers on a blacklist, and includes a means for updating the blacklist transmitted to a server, thereby continuously reducing the risk of fraud.

[0006] A "terminal" is a device that allows a user to make a call and that captures and transmits voice data to a server.

[0007] "Voice data" refers to voice information captured during a call and represented in digital form through signal processing.

[0008] A "voice recognition engine" is a system consisting of software or hardware for converting voice data into text data.

[0009] "Text data" is character string information converted from voice data by a voice recognition engine.

[0010] A "natural language processing algorithm" is a computational method for analyzing text data, understanding grammar, context, meaning, etc., and detecting suspicious patterns.

[0011] A "suspicious pattern" refers to a combination of specific words and contexts that are recognized as signs of specialized fraud.

[0012] "Visual and audio alerts" means visual and audio notifications intended to warn the user when a suspicious pattern is detected.

[0013] A "notification" is a warning message sent to the devices of pre-registered family members or guardians when a suspicious pattern is detected.

[0014] A "blacklist" is a list that registers and manages phone numbers suspected of being fraudulent.

[0015] "Interface" refers to the screen and input means that users use to operate the system.

[0016] "Server" is a central processing unit for receiving voice data, performing voice recognition, natural language processing, sending notifications, and blacklist management. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8]FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0018] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0019] First, the terms used in the following description will be explained.

[0020] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0021] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0022] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0023] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0024] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0025] [First embodiment]

[0026] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0027] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0028] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0029] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0030] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0031] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0032] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0033] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0034] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0035] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0036] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0037] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0038] The present invention is a special fraud prevention system for elderly people, which functions with the cooperation of a terminal, a server, and a user. Specific embodiments of the present invention will be described below.

[0039] 1. Initial device setup

[0040] A special fraud prevention application is installed on the user's device. This application has the function of transmitting the audio of calls in real time to a server. When the user starts the application for the first time, they register the contact information of their family members or guardians and perform the initial settings for the blacklist.

[0041] 2. Processing at the start of a call

[0042] Audio capture and transmission

[0043] The device starts capturing the call as soon as it is initiated, and the audio data is then appropriately serialized and streamed to the server in real time.

[0044] 3. Real-time audio analysis on the server

[0045] Receiving voice data and recognizing voice

[0046] The server receives the voice data sent from the terminal and converts the received voice data into text data using a voice recognition engine.

[0047] Natural language processing and suspicious pattern detection

[0048] The converted text data is then analyzed using natural language processing (NLP) algorithms, which detect specific phrases and contextual patterns that may be indicative of fraud. If a suspicious pattern is detected, a warning flag is raised.

[0049] 4. Warnings and Notifications

[0050] Alert the user

[0051] If a warning flag is raised, the server will send a visual and audio alert to the device to make the user aware of the potential fraud and alert them to it.

[0052] Examples:

[0053] When a suspicious pattern is detected, a pop-up message will appear saying, "This is a potential fraudulent call. Please be careful," and an audio warning will also be issued.

[0054] Notification to family and guardians

[0055] Additionally, the server sends a notification to pre-registered family members or guardians, stating that the user's call may contain suspicious content.

[0056] Examples:

[0057] A message will be sent to the family member's smartphone saying, "User's call may be fraudulent. Please check immediately."

[0058] 5. Post-call processing

[0059] Generate summaries of call data

[0060] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent.

[0061] Call data storage and management

[0062] The generated reports are stored in the user database so that users and their families can review them later.

[0063] 6. Blacklist Management

[0064] Viewing and Editing the Blacklist

[0065] Users can view, add, remove, or edit the blacklisted phone numbers through the app's interface, which is then sent to the server and stored in a database.

[0066] Examples:

[0067] Users can add newly received phone numbers they suspect to be fraudulent to a blacklist, and the changes are immediately reflected on the server.

[0068] 7. Data Security and Privacy

[0069] Data encryption and protection

[0070] All data sent between the server and the device is protected using a powerful encryption algorithm, reducing the risk of personal information and call content being leaked to third parties.

[0071] User-controlled data

[0072] Users can manage their own call history and data usage settings within the app, allowing them to use the system while ensuring their privacy.

[0073] As described above, the present invention can significantly reduce the risk of seniors becoming victims of fraud by integrating multiple functions, such as real-time call content analysis, a warning system, and blacklist management, while providing a user-friendly interface for easy use.

[0074] The processing flow will be explained below.

[0075] Processing at the start of a call

[0076] Step 1: Capture the call audio (on your device)

[0077] 1. Detect the start of a call.

[0078] 2. The system will begin the process of capturing the audio during the call.

[0079] 3. Convert the captured audio data into an appropriate format (e.g., buffer).

[0080] Step 2: Sending audio data (terminal)

[0081] 1. Serialize the converted audio data.

[0082] 2. Stream audio data to the server in real time.

[0083] Real-time analysis of voice data

[0084] Step 3: Receiving audio data (server)

[0085] 1. Receive the audio data sent from the device.

[0086] 2. Deserialize the received audio data into the appropriate format.

[0087] Step 4: Speech Recognition (Server)

[0088] 1. Pass the deserialized voice data to the speech recognition engine.

[0089] 2. Convert the audio data into text data.

[0090] Step 5: Natural Language Processing (Server)

[0091] 1. The converted text data is fed into a natural language processing (NLP) algorithm.

[0092] 2. Analyze the text using NLP algorithms to detect suspicious patterns (indicators of fraud).

[0093] 3. Raise a warning flag if a suspicious pattern is detected.

[0094] Handling notifications and alerts

[0095] Step 6: Sending alerts to users (server)

[0096] 1. If a warning flag is raised, send a visual and audio alert to the user's device.

[0097] 2. The alert will include the message "Suspicious call detected."

[0098] Step 7: Notify family and guardians (server)

[0099] 1. If a warning flag is raised, a notification will be sent to the smartphones of pre-registered family members or guardians.

[0100] 2. The notification will include a warning that "your call may contain suspicious content."

[0101] Post-call processing

[0102] Step 8: Generate summary of call data (server)

[0103] 1. Once the call is complete, generate a call summary report.

[0104] 2. The report should include a summary of the call, any suspicious patterns detected, and a history of notifications sent.

[0105] Step 9: Storing call data (server)

[0106] 1. The generated call summary is saved in the user database.

[0107] 2. The saved data will be available for review by the user or family at a later date.

[0108] Blacklist Management

[0109] Step 10: View and Edit Blacklist (User)

[0110] 1. The user accesses the app's blacklist management screen.

[0111] 2. View the current blacklist.

[0112] 3. The user performs an edit operation, such as adding a new phone number or deleting an existing number.

[0113] Step 11: Update the Blacklist (Server)

[0114] 1. When a user edits the blacklist, the updates are sent to the server.

[0115] 2. The server saves the new blacklist contents in the database.

[0116] 3. The updated blacklist will be used the next time you receive a call.

[0117] Data Security and Privacy

[0118] Step 12: Encrypt and Secure Your Data (Server)

[0119] 1. Protect data sent between the server and the device with a strong encryption algorithm.

[0120] 2. The encryption technology used prevents third parties from eavesdropping or tampering with the data.

[0121] Step 13: User-managed data (User)

[0122] 1. Users can manage their call history and data usage settings within the app.

[0123] 2. You can take actions to ensure privacy, such as deleting data and changing settings.

[0124] Example 1

[0125] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0126] Special frauds targeting the elderly are on the rise, and traditional prevention systems have struggled to detect fraud in real time or respond quickly. Furthermore, it is important to not only detect and warn of signs of fraud, but also to record the contents of calls so that they can be reviewed later. There is also a need for systems that allow users to easily manage blacklists and ensure data safety and privacy.

[0127] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0128] In this invention, the server includes: means for receiving voice data transmitted from the terminal in real time and converting the voice data into text data using a voice recognition engine; means for analyzing the converted text data using a natural language processing algorithm to detect suspicious patterns; means for sending visual and audio alerts to the user's terminal when a suspicious pattern is detected; means for sending notifications to the terminals of pre-registered family members or guardians when a suspicious pattern is detected; and means for generating a summary report of the call content after the call ends and storing it in a user database. This enables the detection and prevention of fraud against elderly people in real time, and allows the content of the call to be reviewed later, enabling more effective fraud prevention. It also allows users to easily manage their blacklist, ensuring data security and privacy.

[0129] "Terminal" means a device used by a user to access the special fraud prevention system.

[0130] A "server" is a central computer system that receives and processes data sent from terminals.

[0131] "Audio data" refers to audio information of the contents of a call captured from a terminal.

[0132] A "voice recognition engine" is a software or hardware technology that converts voice data into text data.

[0133] "Text data" is character string information converted from voice data by a voice recognition engine.

[0134] A "natural language processing algorithm" is a computer program that analyzes text data and detects specified patterns or intent.

[0135] A "suspicious pattern" is a combination of specific words, phrases, or context that indicates possible fraud.

[0136] "Visual and audio alerts" are screen displays and audio notifications that notify the user of the detection of a suspicious pattern.

[0137] "Family and guardians" are trusted contacts registered by the user who will receive notifications in the event of an emergency.

[0138] "Notifications" are warning messages sent to family members or guardians when suspicious patterns are detected.

[0139] A "summary report" is a document generated after a call ends that includes a summary of the call, any suspicious pattern detections, and a history of alerts and notifications.

[0140] A "blacklist" is a list where users register phone numbers that may be fraudulent.

[0141] "Database" means a structured data storage for managing information collected and stored by a system.

[0142] The present invention is a special fraud prevention system for elderly people. This system works with the cooperation of terminals, a server, and users.

[0143] 1. Initial device setup

[0144] The user installs a special fraud prevention application on their device. This application has the function of capturing audio data while the user is on a call and sending it to a server in real time. After installation, the user launches the application for the first time and registers the contact information of their family or guardian. They also perform initial setup to add phone numbers that may be fraudulent to a blacklist. This completes the basic preparations for fraud prevention.

[0145] 2. Processing at the start of a call

[0146] The device starts capturing the audio data of the call as soon as the user starts the call, then the captured audio data is serialized into protocol buffers (Protobuf) or JSON format and streamed to the server in real time.

[0147] 3. Real-time audio analysis on the server

[0148] The server receives voice data sent from the device in real time. The received voice data is converted into text data using a speech recognition engine such as Google Cloud Speech-to-Text. The text data is then analyzed using natural language processing (NLP) algorithms (e.g., spaCy or BERT). The NLP algorithms detect specific phrases and contextual patterns that may be indicative of fraud, and if suspicious patterns are detected, a warning flag is raised.

[0149] 4. Warnings and Notifications

[0150] If a warning flag is raised, the server sends visual and audio alerts to the device, allowing the user to recognize and be on guard against possible fraud. For example, a pop-up message or audio alert may appear on the device stating, "This call may be fraudulent. Please be careful." The server also sends warning notifications to the devices of family members or guardians who the user has registered in advance. This notification includes the message, "This user's call may be fraudulent. Please check immediately."

[0151] 5. Post-call processing

[0152] Once the call is completed, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent. The report is then stored in the user database for later review by the user or family.

[0153] 6. Blacklist Management

[0154] The application interface allows users to add, delete, and edit the blacklist phone numbers, allowing users to add newly received phone numbers that they believe to be fraudulent to the blacklist. The edited blacklist is immediately sent to the server and stored in the database.

[0155] 7. Data Security and Privacy

[0156] All data is protected using strong encryption algorithms such as AES-256 when transmitted between the server and the device, reducing the risk of personal information and call content being leaked to third parties. Users can also manage their call history and data usage settings within the app, ensuring privacy while using the system.

[0157] Examples of specific examples and prompts

[0158] Examples:

[0159] A special fraud prevention app installed on the device automatically sends the contents of the call to a server.

[0160] The server analyzes the call content in real time and detects suspicious patterns.

[0161] If detected, an alert will be sent to you and your family.

[0162] Generate and save detailed reports after the call ends.

[0163] Example prompt sentence:

[0164] "Please explain the voice analysis system to prevent seniors from falling victim to fraud."

[0165] "Please explain in detail how your system analyzes call content in real time to prevent fraud."

[0166] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0167] Step 1:

[0168] Install the application on your device.

[0169] The user installs a special fraud prevention application on the device. After installation is complete, the application is launched for the first time, and the user registers the contact information of family members or guardians and adds phone numbers that may be used for fraud to a blacklist. These settings are sent from the device to the server and saved in a database.

[0170] Input: Application installations, contact information, blacklist information

[0171] Output: Configuration data sent to the server

[0172] Step 2:

[0173] Capture the contents of the call.

[0174] When a user initiates a call, the device starts capturing the call audio in real time, and the captured audio data is serialized and sent to the server in streaming format.

[0175] Input: Call audio

[0176] Output: Serialized audio data

[0177] Step 3:

[0178] Convert the audio data into text data.

[0179] The server receives the voice data sent from the terminal and converts the voice data into text data using a voice recognition engine (e.g., Google Cloud Speech-to-Text).

[0180] Input: Serialized audio data

[0181] Output: Text data

[0182] Step 4:

[0183] Analyze text data.

[0184] The server then analyzes the converted text data using natural language processing (NLP) algorithms (e.g., spaCy or BERT) to detect specific phrases and contextual patterns and look for signs of fraud.

[0185] Input: Text data

[0186] Output: Detection result (whether or not there is a suspicious pattern)

[0187] Step 5:

[0188] Send a warning to the user.

[0189] If the server detects a suspicious pattern, it will send a visual and audio alert to the device. The user's device will display a pop-up message saying, "This is a potential fraudulent call. Please be careful," and will also play an audio alert.

[0190] Input: Detection result (whether or not there is a suspicious pattern)

[0191] Output: Visual and audio alerts to user terminal

[0192] Step 6:

[0193] Send notifications to family members and guardians.

[0194] If the server detects a suspicious pattern, it sends a notification to the device of a pre-registered family member or guardian saying, "The user's call may be fraudulent. Please check immediately."

[0195] Input: Detection result (whether or not there is a suspicious pattern)

[0196] Output: Notification to family and guardians

[0197] Step 7:

[0198] Generate a summary report of the call.

[0199] After the call is completed, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent.

[0200] Input: Call text data and detection results

[0201] Output: Summary report

[0202] Step 8:

[0203] Manage blacklists.

[0204] The application's interface allows users to add, remove, and edit potentially fraudulent phone numbers to the blacklist, which is then sent to the server and stored in a database.

[0205] Input: User edits blacklist information

[0206] Output: Updated blacklist information

[0207] Step 9:

[0208] Ensure data security and privacy.

[0209] All data sent between the server and the device is protected using strong encryption algorithms such as AES-256, reducing the risk of personal information and call content being leaked to third parties.

[0210] Input: All data sent and received

[0211] Output: Encrypted data

[0212] (Application example 1)

[0213] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0214] Elderly people and store employees are at high risk of fraud, especially phone fraud. Employees are required to detect suspicious patterns during calls and respond quickly, but current methods make it difficult to provide real-time warnings and notifications, preventing effective prevention. Therefore, there is a need for a system that can detect signs of fraud during voice calls in real time, immediately warn employees, and notify managers.

[0215] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0216] In this invention, the server includes means for receiving voice data transmitted from the terminal in real time and converting the voice data into text data using a voice recognition engine, means for analyzing the converted text data using a natural language processing algorithm to detect suspicious patterns, means for sending visual and audio alerts to the user's terminal when a suspicious pattern is detected, means for capturing voice data during calls received by employees of a physical store based on an application example of the technology and transmitting it to the server for analysis, means for sending a notification to an administrator terminal when a suspicious pattern is detected, and means for generating a summary report of the call content after the call is ended and storing it on the server, thereby enabling real-time detection of signs of fraud and emergency response.

[0217] "Terminal" refers to the communication equipment used by the user, which is a device that captures and transmits audio data.

[0218] "Voice Data" refers to digital information converted from sound wave signals captured during a call and transmitted in real time to a server.

[0219] "Real-time" means that processing occurs nearly simultaneously with the actual event, with minimal delay.

[0220] A "speech recognition engine" is a software or hardware system for converting voice data into text data.

[0221] "Text data" refers to character information converted by a voice recognition engine, and represents the content of voice data as a string of characters.

[0222] A "natural language processing algorithm" is a computer program that analyzes text data and understands its grammatical structure and meaning.

[0223] A "suspicious pattern" is a specific pattern of words or context that indicates possible fraud and is defined based on historical data.

[0224] A "visual alert" is a visual warning message that appears on a user's device to draw attention when a suspicious pattern is detected.

[0225] "Audio alert" is an audio warning signal intended to audibly alert the user.

[0226] "Administrator" refers to a person in charge of system operation and management in an organization such as a physical store.

[0227] A "call summary report" is a report generated after a call ends that includes the key points of the call and any suspicious pattern detections.

[0228] A "blacklist" is a list of potentially fraudulent phone numbers maintained by the system.

[0229] The present invention is a special fraud prevention system, and a specific embodiment will be described that allows employees in physical stores to quickly detect and respond to fraud risks while on the phone. This system functions through the cooperation of terminals, a server, and users (employees and managers). The operation of each element and their interactions will be described in detail below.

[0230] 1. Initial device setup

[0231] A special fraud prevention application is installed on the user's (employee's) device. This application has a function to transmit the audio of a call to a server in real time. When the user starts the application for the first time, they register the administrator's contact information and perform the initial settings for the blacklist.

[0232] 2. Processing at the start of a call

[0233] Audio capture and transmission

[0234] The device starts capturing the call as soon as it is initiated, and the audio data is then appropriately serialized and streamed to the server in real time.

[0235] 3. Real-time audio analysis on the server

[0236] Receiving voice data and recognizing voice

[0237] The server receives the voice data sent from the device, and converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0238] Natural language processing and suspicious pattern detection

[0239] The converted text data is then analyzed using natural language processing (NLP) algorithms (e.g., SpaCy, nltk). NLP algorithms detect specific phrases and contextual patterns that may be indicative of fraud. If a suspicious pattern is detected, a warning flag is raised.

[0240] 4. Warnings and Notifications

[0241] Alert the user

[0242] If a warning flag is raised, the server will send a visual and audio alert to the terminal, which will alert the user (employee) to the possibility of fraud and bring it to their attention.

[0243] Examples:

[0244] When a suspicious pattern is detected, a pop-up message will appear saying, "This is a potential fraudulent call. Please be careful," and an audio warning will also be issued.

[0245] Administrator Notification

[0246] In addition, the server sends a notification to the device of a pre-registered administrator, stating that the user's (employee's) call may contain suspicious content.

[0247] Examples:

[0248] A message is sent to the administrator's smartphone saying, "An employee's call may be fraudulent. Please check immediately."

[0249] 5. Post-call processing

[0250] Generate summaries of call data

[0251] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent.

[0252] Call data storage and management

[0253] The generated reports are stored in the user database so that users (employees) and administrators can review them later.

[0254] 6. Blacklist Management

[0255] Viewing and Editing the Blacklist

[0256] Users (employees) can use the app interface to view, add, delete, and edit the blacklisted phone numbers, which are then sent to the server and stored in a database.

[0257] Examples:

[0258] Users (employees) can add newly received phone numbers that they believe to be fraudulent to the blacklist, and the changes are immediately reflected on the server.

[0259] 7. Data Security and Privacy

[0260] Data encryption and protection

[0261] All data sent between the server and the device is protected using a powerful encryption algorithm, reducing the risk of personal information and call content being leaked to third parties.

[0262] User-controlled data

[0263] Users (employees) can manage their own call history and data usage settings within the app, allowing them to use the system while ensuring privacy.

[0264] Prompt Sentence Examples

[0265] "I want to develop a system that detects potential fraud for employees taking customer orders over the phone. This system would analyze the audio during the call in real time and alert employees and managers if it detects signs of fraud. Please write code using Python and PyAudio to capture audio data and send it to a server. The server would use a speech recognition and natural language processing engine. Please also include a notification function if a suspicious pattern is detected."

[0266] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0267] Step 1:

[0268] The user (employee) starts the special fraud prevention application on the terminal and performs the initial setup. This requires registering the administrator's contact information and initial setup of the blacklist. This inputs basic information into the terminal for the system to operate properly.

[0269] Input: Administrator contact information, blacklist information

[0270] Output: Device with initial setup completed

[0271] Step 2:

[0272] When a call is initiated, the device begins capturing audio data through the microphone, which is then serialized and streamed to the server in real time.

[0273] Input: Voice data during a call

[0274] Output: Real-time audio data sent to the server

[0275] Step 3:

[0276] The server receives the voice data sent from the device in real time and converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0277] Input: Real-time audio data

[0278] Output: Text data converted by the speech recognition engine

[0279] Step 4:

[0280] The server then analyzes the converted text data using natural language processing (NLP) algorithms (e.g., SpaCy, nltk) to detect specific phrases and contextual patterns that may be indicative of fraud.

[0281] Input: Text data

[0282] Output: Suspicious pattern detection results

[0283] Step 5:

[0284] If a suspicious pattern is detected, the server sends visual and audio alerts to the user's (employee's) device, making them aware of possible fraud and alerting them to the situation.

[0285] Input: Suspicious pattern detection result

[0286] Output: Visual and audio alerts sent to the user's device

[0287] Step 6:

[0288] If the server detects a suspicious pattern, it also sends a notification to the device of a pre-registered administrator, stating that the call content may contain suspicious content.

[0289] Input: Suspicious pattern detection result

[0290] Output: Notification sent to administrator's terminal

[0291] Step 7:

[0292] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent.

[0293] Input: Text data from calls, suspicious pattern detection results, history of alerts and notifications sent

[0294] Output: Call summary report

[0295] Step 8:

[0296] The generated summary report is stored in the server's user database, where users (employees) and managers can review the report at a later time.

[0297] Input: Call Summary Report

[0298] Output: Saved summary report

[0299] Step 9:

[0300] Users (employees) can add new phone numbers to the blacklist, delete them, or edit them. The edited blacklist is sent from the terminal to the server and stored in the database.

[0301] Input: Blacklist edit information

[0302] Output: Updated blacklist

[0303] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0304] The present invention combines an emotion engine with a special fraud prevention system targeted at elderly people, and functions with the cooperation of a terminal, a server, and a user. Specific embodiments of the present invention will be described below.

[0305] 1. Initial device setup

[0306] A special fraud prevention application is installed on the user's device. This application has the function of transmitting the voice during a call to a server in real time and the function of recognizing the user's emotions using an emotion engine. When the user starts the application for the first time, they register the contact information of their family or guardian and perform the initial settings for the blacklist.

[0307] 2. Processing at the start of a call

[0308] Audio capture and transmission

[0309] The device starts capturing the call as soon as it is initiated, and the audio data is then appropriately serialized and streamed to the server in real time.

[0310] 3. Real-time voice analysis and emotion recognition on the server

[0311] Receiving voice data and recognizing voice

[0312] The server receives the voice data sent from the terminal and converts the received voice data into text data using a voice recognition engine.

[0313] Natural language processing and suspicious pattern detection

[0314] The converted text data is then analyzed using natural language processing (NLP) algorithms, which detect specific phrases and contextual patterns that may be indicative of fraud. If a suspicious pattern is detected, a warning flag is raised.

[0315] Emotion recognition by emotion engine

[0316] In parallel, the voice data is fed into an emotion engine to recognize the user's emotions. The emotion engine analyzes voice characteristics such as tone, pitch, and speed to identify the user's emotional state.

[0317] 4. Warnings and Notifications

[0318] Alert the user

[0319] When a warning flag is raised, the server sends visual and audio alerts to the device, whose content and format are adaptively changed based on the user's emotions as recognized by the emotion engine.

[0320] Examples:

[0321] If a suspicious pattern is detected and the user expresses anxiety or fear, a highlighted alert will be displayed stating, "Urgent attention required. Possible scam."

[0322] Notification to family and guardians

[0323] In addition, the server sends a notification to the device of a pre-registered family member or guardian, informing them that the user's call may contain suspicious content and including the user's emotional information as recognized by the emotion engine.

[0324] Examples:

[0325] A message will be sent to the family member's smartphone saying, "A user has made a potentially fraudulent call and we are very concerned. Please check immediately."

[0326] 5. Post-call processing

[0327] Generate summaries of call data

[0328] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, the perceived emotional state, and a history of any alerts or notifications sent.

[0329] Call data storage and management

[0330] The generated reports are stored in the user database so that users and their families can review them later.

[0331] 6. Blacklist Management

[0332] Viewing and Editing the Blacklist

[0333] Users can view, add, remove, or edit the blacklisted phone numbers through the app's interface, which is then sent to the server and stored in a database.

[0334] Examples:

[0335] Users can add newly received phone numbers they suspect to be fraudulent to a blacklist, and the changes are immediately reflected on the server.

[0336] 7. Data Security and Privacy

[0337] Data encryption and protection

[0338] All data sent between the server and the device is protected using a powerful encryption algorithm, reducing the risk of personal information and call content being leaked to third parties.

[0339] User-controlled data

[0340] Users can manage their own call history and data usage settings within the app, allowing them to use the system while ensuring their privacy.

[0341] As described above, this invention can significantly reduce the risk of seniors becoming victims of fraud by integrating real-time call content analysis, a warning system, and emotion recognition. It also provides a user-friendly interface for easy use.

[0342] The processing flow will be explained below.

[0343] Processing at the start of a call

[0344] Step 1: Capture the call audio (on your device)

[0345] 1. Detect the start of a call.

[0346] 2. The system will begin the process of capturing the audio during the call.

[0347] 3. Convert the captured audio data into an appropriate format (e.g., buffer).

[0348] Step 2: Sending audio data (terminal)

[0349] 1. Serialize the converted audio data.

[0350] 2. Stream audio data to the server in real time.

[0351] Real-time analysis of voice data and emotion recognition

[0352] Step 3: Receiving audio data (server)

[0353] 1. Receive the audio data sent from the device.

[0354] 2. Deserialize the received audio data into the appropriate format.

[0355] Step 4: Speech Recognition (Server)

[0356] 1. Pass the deserialized voice data to the speech recognition engine.

[0357] 2. Convert the audio data into text data.

[0358] Step 5: Analysis using natural language processing (NLP) algorithms (server)

[0359] 1. The converted text data is fed into the NLP algorithm.

[0360] 2. Analyze the text using NLP algorithms to detect suspicious patterns (indicators of fraud).

[0361] 3. Raise a warning flag if a suspicious pattern is detected.

[0362] Step 6: Emotion recognition by the emotion engine (server)

[0363] 1. Input the voice data into the emotion engine.

[0364] 2. The emotion engine analyzes the tone, pitch, and rate of speech to identify the user's emotional state.

[0365] 3. The emotional state is recognized and recorded in a database.

[0366] Handling notifications and alerts

[0367] Step 7: Sending alerts to users (server)

[0368] 1. If a warning flag is raised, the server sends a visual and audio alert to the user terminal.

[0369] 2. The content and format of the alerts are adaptively changed based on the user's emotions recognized by the emotion engine.

[0370] Examples:

[0371] If anxiety or fear is detected, a highlighted alert will appear saying, "Urgent attention needed. Possible scam."

[0372] Step 8: Sending notifications to family members and guardians (server)

[0373] 1. If a warning flag is raised, a notification will be sent to the device of a pre-registered family member or guardian.

[0374] 2. The notification will include information about potentially suspicious content in the user's call and the perceived emotional information.

[0375] Examples:

[0376] A message will be sent to family members' smartphones saying, "We are very concerned that a user has made a potentially fraudulent call. Please check immediately."

[0377] Post-call processing

[0378] Step 9: Generate summary of call data (server)

[0379] 1. Once the call is complete, generate a call summary report.

[0380] 2. The report should include a summary of the call, any suspicious patterns detected, any emotional states recognized, and a history of notifications sent.

[0381] Step 10: Storing call data (server)

[0382] 1. The generated call summary is saved in the user database.

[0383] 2. The saved data will be available for review by the user or family at a later date.

[0384] Blacklist Management

[0385] Step 11: View and Edit Blacklist (User)

[0386] 1. The user accesses the app's blacklist management screen.

[0387] 2. View the current blacklist.

[0388] 3. User can add new phone numbers, delete existing numbers and perform edit operations.

[0389] Step 12: Update the Blacklist (Server)

[0390] 1. When a user edits the blacklist, the updates are sent to the server.

[0391] 2. The server saves the new blacklist contents in the database.

[0392] 3. The updated blacklist will be used the next time you receive a call.

[0393] Data Security and Privacy

[0394] Step 13: Encrypt and Secure Your Data (Server)

[0395] 1. Protect data sent between the server and the device with a strong encryption algorithm.

[0396] 2. The encryption technology used prevents third parties from eavesdropping or tampering with the data.

[0397] Step 14: User-managed data (User)

[0398] 1. Users can manage their call history and data usage settings within the app.

[0399] 2. You can take actions to ensure privacy, such as deleting data and changing settings.

[0400] Example 2

[0401] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0402] Special frauds targeting the elderly have been increasing in recent years, making countermeasures urgently needed. The elderly are particularly susceptible to the sophisticated tactics of fraudsters, and the damage they cause affects their families and society as a whole. Furthermore, existing fraud prevention systems lack the accuracy to detect suspicious patterns and have difficulty responding in real time. Furthermore, they lack appropriate alert functions that take into account the user's emotional state, which can prevent the elderly from responding appropriately in emergencies.

[0403] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving voice data transmitted from a terminal in real time and converting the voice data into text data using a voice recognition module; means for analyzing the converted text data using a natural language analysis algorithm to detect suspicious patterns; means for identifying the user's emotional state using an emotion analysis engine when a suspicious pattern is detected; means for sending adaptive visual and audio alerts to the terminal based on the user's emotional state; and means for sending notifications to pre-registered contacts when a suspicious pattern is detected. This significantly reduces the risk of elderly people becoming victims of special fraud and enables rapid response in real time. Furthermore, alerts that take emotional state into account allow users to take appropriate action according to the situation.

[0404] "Audio data" is data that represents an audio signal transmitted from a terminal in digital form.

[0405] A "voice recognition module" is software or hardware for converting voice data into text data.

[0406] "Text data" refers to data that includes character information converted by a voice recognition module.

[0407] A "natural language analysis algorithm" is a set of techniques or processes that analyze text data to detect specific words, phrases, and contexts.

[0408] A "suspicious pattern" is a pattern of specific words, context, or behavior that suggests fraud or wrongdoing.

[0409] An "emotion analysis engine" is a system that recognizes and identifies a user's emotional state from voice and text.

[0410] A "visual alert" is a visual display of a warning message or reminder that is displayed on a device's display.

[0411] A "voice alert" is a warning message or warning alert issued by the device.

[0412] "Registered Contacts" refers to information about family members, guardians, and other emergency contacts that the user has previously registered in the system.

[0413] "Real-time" means that processing and communication occur almost simultaneously with real time.

[0414] "Terminal" refers to a communication device or computer system used by a user, and is a device that constitutes part of this system.

[0415] "Server" refers to a central computer system that receives and processes data from terminals.

[0416] The present invention combines an emotion analysis function with a special fraud prevention system targeted at elderly people, and is a system that functions with the cooperation of terminals, servers, and users. Specific embodiments of the present invention are described in detail below.

[0417] A special fraud prevention application is installed on the user's device. This application has built-in functions for transmitting voice to a server in real time and for recognizing the user's emotions using an emotion analysis engine. When the application is first launched, the user registers the contact information of family members or guardians and sets up a blacklist.

[0418] 1. Initial device setup

[0419] Users install a special fraud prevention application on their device and enter the contact information of their family members or guardians and their blacklist within the application. Once registration is complete, this data is sent to a server and stored in a database.

[0420] 2. When the call starts

[0421] When a user initiates a call, the device automatically starts capturing the call. During the call, audio data is sent to the server in real time, where it is properly serialized and streamed using WebSockets.

[0422] 3. Real-time analysis by the server

[0423] The server receives the voice data sent from the device in real time and converts it into text data using the Google Speech-to-Text API. The converted text data is analyzed using a natural language analysis algorithm (e.g., SpaCy) to detect suspicious patterns. If a suspicious pattern is detected, a warning flag is raised. In parallel, the voice data is sent to an emotion analysis engine to identify the user's emotional state.

[0424] For example, if a user makes a call saying, "Please transfer money," the natural language analysis algorithm will detect potentially fraudulent keywords such as "money" and "transfer" and raise a warning flag.

[0425] 4. Sending alerts and notifications

[0426] When the server receives a warning flag, it sends a visual and audio alert to the user's device. The content of the alert changes adaptively based on the user's emotions, as recognized by an emotion analysis engine. For example, if the user expresses anxiety or fear, a highlighted alert will be displayed stating, "Urgent attention required. Possible fraud." The server also sends notifications to pre-registered contacts to inform the user of a suspicious call.

[0427] For example, a message may be sent to a family member's smartphone saying, "We are very concerned that a user has made a potentially fraudulent call. Please check immediately."

[0428] 5. Post-call processing

[0429] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, the perceived emotional state, and a history of any alerts or notifications sent. The report is stored in the user database for later review by the user or family members.

[0430] 6. Blacklist Management

[0431] Through the application interface, users can view, add, delete and edit the blacklist, and these changes are sent to the server and stored in a database.

[0432] For example, when a user adds a newly received phone number that they believe to be fraudulent to the blacklist, the changes are immediately reflected on the server.

[0433] 7. Data Security and Privacy

[0434] Data transmitted between the server and the device is protected using strong encryption algorithms such as AES-256. In addition, users can manage their call history and data usage settings within the app, ensuring their privacy and allowing them to use the system with peace of mind.

[0435] The system significantly reduces the risk of seniors becoming victims of fraud and enables real-time responses, with adaptive alerts based on emotional state allowing users to take appropriate action quickly.

[0436] Example prompt sentence:

[0437] "I'm thinking about a system to prevent special frauds against the elderly. Could you please give me an overview of the system that analyzes phone calls in real time and issues a warning if there are signs of fraud?"

[0438] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0439] Step 1:

[0440] The user installs the special fraud prevention application on their device. When they first launch it, they enter the contact information and blacklist of their family members or guardians to register them in the application.

[0441] Specifically, the user downloads and installs the application. After launching the application, they enter the necessary contact and blacklist information on the registration screen that appears, and then press the "Save" button. This information is sent to the server and saved in a database.

[0442] Input: User contact information, blacklist information

[0443] Output: Preferences information stored in the server database

[0444] Step 2:

[0445] When a user initiates a call, the device automatically prepares to capture the call content.

[0446] Specifically, the app uses the mobile phone's API to detect when a new call starts, and the moment the call starts, the app automatically launches in the background.

[0447] Input: Call Start Event

[0448] Output: Ready to capture conversation

[0449] Step 3:

[0450] The device captures the audio data of the call and streams it in real time to the server, where it is appropriately serialized and transmitted using WebSockets.

[0451] Specifically, the device stores the call audio in a buffer. Whenever the data in this buffer reaches a certain amount, the audio data is streamed via WebSocket.

[0452] Input: Call audio data

[0453] Output: Serialized audio data sent to the server

[0454] Step 4:

[0455] The server receives the voice data transmitted from the terminal in real time and converts the voice data into text data using a voice recognition module.

[0456] Specifically, when the server receives the voice data, it calls the Google Speech-to-Text API to perform speech recognition, and the results are returned as text data.

[0457] Input: Serialized audio data

[0458] Output: Text data

[0459] Step 5:

[0460] The server then analyzes the converted text data using natural language analysis algorithms to detect suspicious patterns, while a sentiment analysis engine identifies the user's emotional state.

[0461] Specifically, the server analyzes the text data using SpaCy to detect suspicious patterns based on specific keywords and context, while simultaneously sending the audio data to an emotion analysis engine to determine the user's emotional state based on their tone and pitch.

[0462] Input: Text data, audio data

[0463] Output: Presence or absence of suspicious patterns, emotional state

[0464] Step 6:

[0465] If a suspicious pattern is detected, the server sends adaptive visual and audio alerts to the user's device based on their emotional state.

[0466] Specifically, the server checks the warning flag, and if the user shows signs of anxiety or fear, it sends an alert message to the user's device stating, "Urgent attention required. Possible fraud."

[0467] Input: Suspicious pattern detection results, emotional state

[0468] Output: An adaptive alert message sent to the user's device.

[0469] Step 7:

[0470] If a suspicious pattern is detected, the server will also send a notification to pre-registered contacts.

[0471] Specifically, the server will notify the user's family or guardian with a message stating, "The user has made a potentially fraudulent call and is very concerned. Please check immediately."

[0472] Input: Suspicious pattern detection results, emotional state

[0473] Output: Notification message sent to registered contacts

[0474] Step 8:

[0475] Once the call is over, the server generates a summary report of the call and stores it in the user database, including a summary of the call, any suspicious patterns detected, emotional state, and a history of any alerts or notifications sent.

[0476] Specifically, the server generates a summary report based on the full text of the call and emotion recognition data and stores it in a database.

[0477] Input: Full text data of calls, emotion recognition data

[0478] Output: Generated summary report

[0479] Step 9:

[0480] Users can view, add, delete and edit their blacklists, and these changes are sent to the server and stored in the database.

[0481] Specifically, when a user accesses the "Blacklist Management" menu in the app, enters a new phone number, and presses the "Add" button, the number is added to the blacklist. This information is sent to the server and stored in a database.

[0482] Input: User-modified blacklist

[0483] Output: Blacklist information updated in the server database

[0484] (Application example 2)

[0485] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0486] Seniors need protection from fraud and suspicious transactions when shopping in brick-and-mortar stores. However, current systems lack real-time monitoring and emotion recognition in brick-and-mortar stores, putting seniors at high risk of fraud. Furthermore, existing prevention systems lack adaptive warnings that take into account the user's emotional state, preventing appropriate responses in emergencies.

[0487] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0488] In this invention, the server includes means for receiving voice data transmitted from the terminal in real time and converting the voice data into text data using a voice recognition engine, means for analyzing the converted text data using a natural language processing algorithm to detect suspicious patterns, means for sending visual and audio alerts to the user's terminal when a suspicious pattern is detected, means for sending notifications to terminals of pre-registered family members or guardians when a suspicious pattern is detected, means for analyzing the voice data using an emotion recognition engine to identify the user's emotional state, and means for adaptively changing the corresponding alert format when the user expresses anxiety. This allows elderly people to enjoy shopping in physical stores without worrying about fraud or suspicious transactions.

[0489] A "terminal" is a device that captures audio and video data and transmits it to a server in real time.

[0490] "Voice data" refers to data that records voice information such as calls and conversations in digital format.

[0491] "Real-time" is a term that refers to data being processed immediately as it is acquired, without delay.

[0492] A "voice recognition engine" is a general term for software and algorithms that analyze voice data and convert its contents into text data.

[0493] "Text data" refers to character information converted from voice data by a voice recognition engine.

[0494] A "natural language processing algorithm" is a set of computational methods that allow a computer to process human language and analyze its meaning.

[0495] A "suspicious pattern" refers to a combination of specific words, phrases, or context that may indicate fraud or misconduct.

[0496] "Visual and audio alerts" means visual indications and audio notifications intended to alert the user.

[0497] An "emotion recognition engine" is a technology that uses audio and video data to analyze a user's emotional state.

[0498] "Adaptively changing" means automatically adjusting the system's output and behavior according to the situation or conditions.

[0499] "Family or guardian" refers to a person who supports the user and whose contact information has been registered in the system in advance.

[0500] "Alert Format" means a particular method or means of notifying a user of an urgent event.

[0501] The present invention combines an emotion recognition engine with a special fraud prevention system targeted at elderly people. Specific embodiments are described below.

[0502] 1. System Configuration

[0503] Hardware used

[0504] Smartphone

[0505] Smart Glasses

[0506] Software used

[0507] Speech recognition engine: Google Cloud Speech-to-Text API

[0508] Natural Language Processing Algorithm: BERT Model

[0509] Emotion Recognition Engine: Affectiva API

[0510] Communication protocol: WebRTC

[0511] Alerting system: Firebase Cloud Messaging (FCM)

[0512] 2. Specific implementation method of the system

[0513] User device initial settings

[0514] A fraud prevention application is installed on the user's smartphone or smart glasses. This application has the ability to capture audio and video data in real time and send it to a server. When the application is launched for the first time, the user registers the contact information of family members or guardians and performs initial settings for the blacklist.

[0515] Initiating calls and conversations

[0516] As soon as a call or in-store interaction is initiated, the device begins capturing audio and video data, which is serialized in real time and streamed to the server.

[0517] Data analysis on the server

[0518] The server receives the data sent from the device. The voice data is converted to text data using the Google Cloud Speech-to-Text API. The converted text data is analyzed using a natural language processing algorithm using the BERT model to detect suspicious patterns. If a suspicious pattern is detected, a warning flag is raised.

[0519] In parallel, the voice data is fed into an emotion recognition engine using the Affectiva API to analyze the user's emotional state. Based on the detailed analysis, if the user is expressing anxiety or fear, the alert format will be adaptively changed.

[0520] Alerts and Notifications

[0521] If a suspicious pattern is detected, the server will send a visual and audio alert to the device, such as an alert saying "Urgent attention required. Possible fraud."

[0522] Additionally, notifications will be sent to pre-registered family members and guardians on their devices, stating, "A user has made a potentially fraudulent call, and we are very concerned. Please check immediately."

[0523] Post-call processing

[0524] Once the call or interaction is over, the server generates a summary report of the call and stores it in a database for later review by the user or family members, including a summary of the call, any suspicious pattern detections, emotional state, and a history of any alerts or notifications sent.

[0525] Managing the Blacklist

[0526] Users can view, add, remove, and edit the blacklisted phone numbers and contacts through the app's interface, which is then sent to the server and stored in a database.

[0527] 3. Examples and prompts

[0528] Specific examples

[0529] If a scam indicator is detected in a physical store, such as "You need to give us your credit card details right away" or "You need to complete this process now to get a big discount!"

[0530] The user's smart glasses will play a message saying "Urgent attention required. Possible scam" along with an alert sound.

[0531] A notification will be sent to the family member's smartphone saying, "A user is engaged in a potentially fraudulent transaction and we are very concerned. Please check with the store immediately."

[0532] Prompt Sentence Examples

[0533] Input data: Audio data and real-time streaming

[0534] Emotion recognition API: Affectiva

[0535] Natural Language Processing API: Google Cloud Speech-to-Text, BERT model

[0536] process:

[0537] Converts voice data into text and analyzes it for signs of fraud

[0538] Analyze the user's emotional state

[0539] Triggers warnings when it detects suspicious patterns and user concerns

[0540] output:

[0541] Visual and audio alerts on the user's device

[0542] Notifications on family and guardian devices

[0543] As described above, the present invention provides a system that protects elderly people from fraud and suspicious transactions in physical stores, allowing them to enjoy shopping with peace of mind.By recognizing user emotions and providing adaptive warnings, the system can respond quickly and appropriately in emergencies.

[0544] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0545] Step 1:

[0546] When a call or conversation in a physical store is initiated, the device captures audio and video data using its microphone and camera. This data is streamed to a server in real time using the WebRTC protocol. The input is audio and video data, and the output is serialized streaming data.

[0547] Step 2:

[0548] The server receives the voice data sent from the device in real time. The received voice data is input into the Google Cloud Speech-to-Text API, which converts the voice into text data. The input is voice data, and the output is voice transcription data (text data).

[0549] Step 3:

[0550] The server analyzes text data using a natural language processing algorithm based on the BERT model to detect suspicious patterns. The input is text data, and the output is a flag indicating whether a suspicious pattern exists. Specifically, the server performs grammatical analysis of the text data to detect suspicious words and phrases.

[0551] Step 4:

[0552] The server simultaneously analyzes the voice data with an emotion recognition engine using the Affectiva API to identify the user's emotional state. The input is the voice data, and the output is a classification of the user's emotional state (e.g., anxiety, fear, relief, etc.). Specific operations include analyzing the tone, pitch, and speed of the voice.

[0553] Step 5:

[0554] The server sends visual and audio alerts to the device if a suspicious pattern is detected and the user's emotional state indicates anxiety or fear. The input is a suspicious pattern flag and a classification of the emotional state, and the output is a visual and audio alert. Specific behaviors include adaptively changing the content and format of the alert.

[0555] Step 6:

[0556] If a suspicious pattern is detected, the server also sends a notification to the devices of pre-registered family members or guardians. The input is a suspicious pattern flag and the user's emotional state, and the output is a notification message to the family or guardian. Specifically, the notification is sent using Firebase Cloud Messaging (FCM).

[0557] Step 7:

[0558] Once the call or conversation is over, the server generates a summary report of the call and stores it in a database for later review by the user or family. The input is the text data of the call or conversation, the results of suspicious pattern detection, emotional state, and a history of sent alerts and notifications, and the output is a summary report. Specifically, this data is compiled into a single document and stored in the database.

[0559] Step 8:

[0560] Users can use the app's interface to view, add, delete, and edit blacklisted phone numbers and contacts. The input is the user's instructions, and the output is the updated blacklist information. Specifically, after accepting the user's operation, the app sends the updated blacklist data to the server and updates the database.

[0561] The above processing steps enable real-time audio and video data analysis, emotion recognition, suspicious pattern detection, warning systems, and notification systems to work together in an integrated manner, allowing elderly people to enjoy shopping in physical stores with peace of mind.

[0562] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0563] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0564] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0565] [Second embodiment]

[0566] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0567] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0568] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0569] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0570] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0571] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0572] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0573] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0574] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0575] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0576] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0577] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0578] The present invention is a special fraud prevention system for elderly people, which functions with the cooperation of a terminal, a server, and a user. Specific embodiments of the present invention will be described below.

[0579] 1. Initial device setup

[0580] A special fraud prevention application is installed on the user's device. This application has the function of transmitting the audio of calls in real time to a server. When the user starts the application for the first time, they register the contact information of their family members or guardians and perform the initial settings for the blacklist.

[0581] 2. Processing at the start of a call

[0582] Audio capture and transmission

[0583] The device starts capturing the call as soon as it is initiated, and the audio data is then appropriately serialized and streamed to the server in real time.

[0584] 3. Real-time audio analysis on the server

[0585] Receiving voice data and recognizing voice

[0586] The server receives the voice data sent from the terminal and converts the received voice data into text data using a voice recognition engine.

[0587] Natural language processing and suspicious pattern detection

[0588] The converted text data is then analyzed using natural language processing (NLP) algorithms, which detect specific phrases and contextual patterns that may be indicative of fraud. If a suspicious pattern is detected, a warning flag is raised.

[0589] 4. Warnings and Notifications

[0590] Alert the user

[0591] If a warning flag is raised, the server will send a visual and audio alert to the device to make the user aware of the potential fraud and alert them to it.

[0592] Examples:

[0593] When a suspicious pattern is detected, a pop-up message will appear saying, "This is a potential fraudulent call. Please be careful," and an audio warning will also be issued.

[0594] Notification to family and guardians

[0595] Additionally, the server sends a notification to pre-registered family members or guardians, stating that the user's call may contain suspicious content.

[0596] Examples:

[0597] A message will be sent to the family member's smartphone saying, "User's call may be fraudulent. Please check immediately."

[0598] 5. Post-call processing

[0599] Generate summaries of call data

[0600] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent.

[0601] Call data storage and management

[0602] The generated reports are stored in the user database so that users and their families can review them later.

[0603] 6. Blacklist Management

[0604] Viewing and Editing the Blacklist

[0605] Users can view, add, remove, or edit the blacklisted phone numbers through the app's interface, which is then sent to the server and stored in a database.

[0606] Examples:

[0607] Users can add newly received phone numbers they suspect to be fraudulent to a blacklist, and the changes are immediately reflected on the server.

[0608] 7. Data Security and Privacy

[0609] Data encryption and protection

[0610] All data sent between the server and the device is protected using a powerful encryption algorithm, reducing the risk of personal information and call content being leaked to third parties.

[0611] User-controlled data

[0612] Users can manage their own call history and data usage settings within the app, allowing them to use the system while ensuring their privacy.

[0613] As described above, the present invention can significantly reduce the risk of seniors becoming victims of fraud by integrating multiple functions, such as real-time call content analysis, a warning system, and blacklist management, while providing a user-friendly interface for easy use.

[0614] The processing flow will be explained below.

[0615] Processing at the start of a call

[0616] Step 1: Capture the call audio (on your device)

[0617] 1. Detect the start of a call.

[0618] 2. The system will begin the process of capturing the audio during the call.

[0619] 3. Convert the captured audio data into an appropriate format (e.g., buffer).

[0620] Step 2: Sending audio data (terminal)

[0621] 1. Serialize the converted audio data.

[0622] 2. Stream audio data to the server in real time.

[0623] Real-time analysis of voice data

[0624] Step 3: Receiving audio data (server)

[0625] 1. Receive the audio data sent from the device.

[0626] 2. Deserialize the received audio data into the appropriate format.

[0627] Step 4: Speech Recognition (Server)

[0628] 1. Pass the deserialized voice data to the speech recognition engine.

[0629] 2. Convert the audio data into text data.

[0630] Step 5: Natural Language Processing (Server)

[0631] 1. The converted text data is fed into a natural language processing (NLP) algorithm.

[0632] 2. Analyze the text using NLP algorithms to detect suspicious patterns (indicators of fraud).

[0633] 3. Raise a warning flag if a suspicious pattern is detected.

[0634] Handling notifications and alerts

[0635] Step 6: Sending alerts to users (server)

[0636] 1. If a warning flag is raised, send a visual and audio alert to the user's device.

[0637] 2. The alert will include the message "Suspicious call detected."

[0638] Step 7: Notify family and guardians (server)

[0639] 1. If a warning flag is raised, a notification will be sent to the smartphones of pre-registered family members or guardians.

[0640] 2. The notification will include a warning that "your call may contain suspicious content."

[0641] Post-call processing

[0642] Step 8: Generate summary of call data (server)

[0643] 1. Once the call is complete, generate a call summary report.

[0644] 2. The report should include a summary of the call, any suspicious patterns detected, and a history of notifications sent.

[0645] Step 9: Storing call data (server)

[0646] 1. The generated call summary is saved in the user database.

[0647] 2. The saved data will be available for review by the user or family at a later date.

[0648] Blacklist Management

[0649] Step 10: View and Edit Blacklist (User)

[0650] 1. The user accesses the app's blacklist management screen.

[0651] 2. View the current blacklist.

[0652] 3. The user performs an edit operation, such as adding a new phone number or deleting an existing number.

[0653] Step 11: Update the Blacklist (Server)

[0654] 1. When a user edits the blacklist, the updates are sent to the server.

[0655] 2. The server saves the new blacklist contents in the database.

[0656] 3. The updated blacklist will be used the next time you receive a call.

[0657] Data Security and Privacy

[0658] Step 12: Encrypt and Secure Your Data (Server)

[0659] 1. Protect data sent between the server and the device with a strong encryption algorithm.

[0660] 2. The encryption technology used prevents third parties from eavesdropping or tampering with the data.

[0661] Step 13: User-managed data (User)

[0662] 1. Users can manage their call history and data usage settings within the app.

[0663] 2. You can take actions to ensure privacy, such as deleting data and changing settings.

[0664] Example 1

[0665] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0666] Special frauds targeting the elderly are on the rise, and traditional prevention systems have struggled to detect fraud in real time or respond quickly. Furthermore, it is important to not only detect and warn of signs of fraud, but also to record the contents of calls so that they can be reviewed later. There is also a need for systems that allow users to easily manage blacklists and ensure data safety and privacy.

[0667] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0668] In this invention, the server includes: means for receiving voice data transmitted from the terminal in real time and converting the voice data into text data using a voice recognition engine; means for analyzing the converted text data using a natural language processing algorithm to detect suspicious patterns; means for sending visual and audio alerts to the user's terminal when a suspicious pattern is detected; means for sending notifications to the terminals of pre-registered family members or guardians when a suspicious pattern is detected; and means for generating a summary report of the call content after the call ends and storing it in a user database. This enables the detection and prevention of fraud against elderly people in real time, and allows the content of the call to be reviewed later, enabling more effective fraud prevention. It also allows users to easily manage their blacklist, ensuring data security and privacy.

[0669] "Terminal" means a device used by a user to access the special fraud prevention system.

[0670] A "server" is a central computer system that receives and processes data sent from terminals.

[0671] "Audio data" refers to audio information of the contents of a call captured from a terminal.

[0672] A "voice recognition engine" is a software or hardware technology that converts voice data into text data.

[0673] "Text data" is character string information converted from voice data by a voice recognition engine.

[0674] A "natural language processing algorithm" is a computer program that analyzes text data and detects specified patterns or intent.

[0675] A "suspicious pattern" is a combination of specific words, phrases, or context that indicates possible fraud.

[0676] "Visual and audio alerts" are screen displays and audio notifications that notify the user of the detection of a suspicious pattern.

[0677] "Family and guardians" are trusted contacts registered by the user who will receive notifications in the event of an emergency.

[0678] "Notifications" are warning messages sent to family members or guardians when suspicious patterns are detected.

[0679] A "summary report" is a document generated after a call ends that includes a summary of the call, any suspicious pattern detections, and a history of alerts and notifications.

[0680] A "blacklist" is a list where users register phone numbers that may be fraudulent.

[0681] "Database" means a structured data storage for managing information collected and stored by a system.

[0682] The present invention is a special fraud prevention system for elderly people. This system works with the cooperation of terminals, a server, and users.

[0683] 1. Initial device setup

[0684] The user installs a special fraud prevention application on their device. This application has the function of capturing audio data while the user is on a call and sending it to a server in real time. After installation, the user launches the application for the first time and registers the contact information of their family or guardian. They also perform initial setup to add phone numbers that may be fraudulent to a blacklist. This completes the basic preparations for fraud prevention.

[0685] 2. Processing at the start of a call

[0686] The device starts capturing the audio data of the call as soon as the user starts the call, then the captured audio data is serialized into protocol buffers (Protobuf) or JSON format and streamed to the server in real time.

[0687] 3. Real-time audio analysis on the server

[0688] The server receives voice data sent from the device in real time. The received voice data is converted into text data using a speech recognition engine such as Google Cloud Speech-to-Text. The text data is then analyzed using natural language processing (NLP) algorithms (e.g., spaCy or BERT). The NLP algorithms detect specific phrases and contextual patterns that may be indicative of fraud, and if suspicious patterns are detected, a warning flag is raised.

[0689] 4. Warnings and Notifications

[0690] If a warning flag is raised, the server sends visual and audio alerts to the device, allowing the user to recognize and be on guard against possible fraud. For example, a pop-up message or audio alert may appear on the device stating, "This call may be fraudulent. Please be careful." The server also sends warning notifications to the devices of family members or guardians who the user has registered in advance. This notification includes the message, "This user's call may be fraudulent. Please check immediately."

[0691] 5. Post-call processing

[0692] Once the call is completed, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent. The report is then stored in the user database for later review by the user or family.

[0693] 6. Blacklist Management

[0694] The application interface allows users to add, delete, and edit the blacklist phone numbers, allowing users to add newly received phone numbers that they believe to be fraudulent to the blacklist. The edited blacklist is immediately sent to the server and stored in the database.

[0695] 7. Data Security and Privacy

[0696] All data is protected using strong encryption algorithms such as AES-256 when transmitted between the server and the device, reducing the risk of personal information and call content being leaked to third parties. Users can also manage their call history and data usage settings within the app, ensuring privacy while using the system.

[0697] Examples of specific examples and prompts

[0698] Examples:

[0699] A special fraud prevention app installed on the device automatically sends the contents of the call to a server.

[0700] The server analyzes the call content in real time and detects suspicious patterns.

[0701] If detected, an alert will be sent to you and your family.

[0702] Generate and save detailed reports after the call ends.

[0703] Example prompt sentence:

[0704] "Please explain the voice analysis system to prevent seniors from falling victim to fraud."

[0705] "Please explain in detail how your system analyzes call content in real time to prevent fraud."

[0706] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0707] Step 1:

[0708] Install the application on your device.

[0709] The user installs a special fraud prevention application on the device. After installation is complete, the application is launched for the first time, and the user registers the contact information of family members or guardians and adds phone numbers that may be used for fraud to a blacklist. These settings are sent from the device to the server and saved in a database.

[0710] Input: Application installations, contact information, blacklist information

[0711] Output: Configuration data sent to the server

[0712] Step 2:

[0713] Capture the contents of the call.

[0714] When a user initiates a call, the device starts capturing the call audio in real time, and the captured audio data is serialized and sent to the server in streaming format.

[0715] Input: Call audio

[0716] Output: Serialized audio data

[0717] Step 3:

[0718] Convert the audio data into text data.

[0719] The server receives the voice data sent from the terminal and converts the voice data into text data using a voice recognition engine (e.g., Google Cloud Speech-to-Text).

[0720] Input: Serialized audio data

[0721] Output: Text data

[0722] Step 4:

[0723] Analyze text data.

[0724] The server then analyzes the converted text data using natural language processing (NLP) algorithms (e.g., spaCy or BERT) to detect specific phrases and contextual patterns and look for signs of fraud.

[0725] Input: Text data

[0726] Output: Detection result (whether or not there is a suspicious pattern)

[0727] Step 5:

[0728] Send a warning to the user.

[0729] If the server detects a suspicious pattern, it will send a visual and audio alert to the device. The user's device will display a pop-up message saying, "This is a potential fraudulent call. Please be careful," and will also play an audio alert.

[0730] Input: Detection result (whether or not there is a suspicious pattern)

[0731] Output: Visual and audio alerts to user terminal

[0732] Step 6:

[0733] Send notifications to family members and guardians.

[0734] If the server detects a suspicious pattern, it sends a notification to the device of a pre-registered family member or guardian saying, "The user's call may be fraudulent. Please check immediately."

[0735] Input: Detection result (whether or not there is a suspicious pattern)

[0736] Output: Notification to family and guardians

[0737] Step 7:

[0738] Generate a summary report of the call.

[0739] After the call is completed, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent.

[0740] Input: Call text data and detection results

[0741] Output: Summary report

[0742] Step 8:

[0743] Manage blacklists.

[0744] The application's interface allows users to add, remove, and edit potentially fraudulent phone numbers to the blacklist, which is then sent to the server and stored in a database.

[0745] Input: User edits blacklist information

[0746] Output: Updated blacklist information

[0747] Step 9:

[0748] Ensure data security and privacy.

[0749] All data sent between the server and the device is protected using strong encryption algorithms such as AES-256, reducing the risk of personal information and call content being leaked to third parties.

[0750] Input: All data sent and received

[0751] Output: Encrypted data

[0752] (Application example 1)

[0753] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0754] Elderly people and store employees are at high risk of fraud, especially phone fraud. Employees are required to detect suspicious patterns during calls and respond quickly, but current methods make it difficult to provide real-time warnings and notifications, preventing effective prevention. Therefore, there is a need for a system that can detect signs of fraud during voice calls in real time, immediately warn employees, and notify managers.

[0755] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0756] In this invention, the server includes means for receiving voice data transmitted from the terminal in real time and converting the voice data into text data using a voice recognition engine, means for analyzing the converted text data using a natural language processing algorithm to detect suspicious patterns, means for sending visual and audio alerts to the user's terminal when a suspicious pattern is detected, means for capturing voice data during calls received by employees of a physical store based on an application example of the technology and transmitting it to the server for analysis, means for sending a notification to an administrator terminal when a suspicious pattern is detected, and means for generating a summary report of the call content after the call is ended and storing it on the server, thereby enabling real-time detection of signs of fraud and emergency response.

[0757] "Terminal" refers to the communication equipment used by the user, which is a device that captures and transmits audio data.

[0758] "Voice Data" refers to digital information converted from sound wave signals captured during a call and transmitted in real time to a server.

[0759] "Real-time" means that processing occurs nearly simultaneously with the actual event, with minimal delay.

[0760] A "speech recognition engine" is a software or hardware system for converting voice data into text data.

[0761] "Text data" refers to character information converted by a voice recognition engine, and represents the content of voice data as a string of characters.

[0762] A "natural language processing algorithm" is a computer program that analyzes text data and understands its grammatical structure and meaning.

[0763] A "suspicious pattern" is a specific pattern of words or context that indicates possible fraud and is defined based on historical data.

[0764] A "visual alert" is a visual warning message that appears on a user's device to draw attention when a suspicious pattern is detected.

[0765] "Audio alert" is an audio warning signal intended to audibly alert the user.

[0766] "Administrator" refers to a person in charge of system operation and management in an organization such as a physical store.

[0767] A "call summary report" is a report generated after a call ends that includes the key points of the call and any suspicious pattern detections.

[0768] A "blacklist" is a list of potentially fraudulent phone numbers maintained by the system.

[0769] The present invention is a special fraud prevention system, and a specific embodiment will be described that allows employees in physical stores to quickly detect and respond to fraud risks while on the phone. This system functions through the cooperation of terminals, a server, and users (employees and managers). The operation of each element and their interactions will be described in detail below.

[0770] 1. Initial device setup

[0771] A special fraud prevention application is installed on the user's (employee's) device. This application has a function to transmit the audio of a call to a server in real time. When the user starts the application for the first time, they register the administrator's contact information and perform the initial settings for the blacklist.

[0772] 2. Processing at the start of a call

[0773] Audio capture and transmission

[0774] The device starts capturing the call as soon as it is initiated, and the audio data is then appropriately serialized and streamed to the server in real time.

[0775] 3. Real-time audio analysis on the server

[0776] Receiving voice data and recognizing voice

[0777] The server receives the voice data sent from the device, and converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0778] Natural language processing and suspicious pattern detection

[0779] The converted text data is then analyzed using natural language processing (NLP) algorithms (e.g., SpaCy, nltk). NLP algorithms detect specific phrases and contextual patterns that may be indicative of fraud. If a suspicious pattern is detected, a warning flag is raised.

[0780] 4. Warnings and Notifications

[0781] Alert the user

[0782] If a warning flag is raised, the server will send a visual and audio alert to the terminal, which will alert the user (employee) to the possibility of fraud and bring it to their attention.

[0783] Examples:

[0784] When a suspicious pattern is detected, a pop-up message will appear saying, "This is a potential fraudulent call. Please be careful," and an audio warning will also be issued.

[0785] Administrator Notification

[0786] In addition, the server sends a notification to the device of a pre-registered administrator, stating that the user's (employee's) call may contain suspicious content.

[0787] Examples:

[0788] A message is sent to the administrator's smartphone saying, "An employee's call may be fraudulent. Please check immediately."

[0789] 5. Post-call processing

[0790] Generate summaries of call data

[0791] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent.

[0792] Call data storage and management

[0793] The generated reports are stored in the user database so that users (employees) and administrators can review them later.

[0794] 6. Blacklist Management

[0795] Viewing and Editing the Blacklist

[0796] Users (employees) can use the app interface to view, add, delete, and edit the blacklisted phone numbers, which are then sent to the server and stored in a database.

[0797] Examples:

[0798] Users (employees) can add newly received phone numbers that they believe to be fraudulent to the blacklist, and the changes are immediately reflected on the server.

[0799] 7. Data Security and Privacy

[0800] Data encryption and protection

[0801] All data sent between the server and the device is protected using a powerful encryption algorithm, reducing the risk of personal information and call content being leaked to third parties.

[0802] User-controlled data

[0803] Users (employees) can manage their own call history and data usage settings within the app, allowing them to use the system while ensuring privacy.

[0804] Prompt Sentence Examples

[0805] "I want to develop a system that detects potential fraud for employees taking customer orders over the phone. This system would analyze the audio during the call in real time and alert employees and managers if it detects signs of fraud. Please write code using Python and PyAudio to capture audio data and send it to a server. The server would use a speech recognition and natural language processing engine. Please also include a notification function if a suspicious pattern is detected."

[0806] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0807] Step 1:

[0808] The user (employee) starts the special fraud prevention application on the terminal and performs the initial setup. This requires registering the administrator's contact information and initial setup of the blacklist. This inputs basic information into the terminal for the system to operate properly.

[0809] Input: Administrator contact information, blacklist information

[0810] Output: Device with initial setup completed

[0811] Step 2:

[0812] When a call is initiated, the device begins capturing audio data through the microphone, which is then serialized and streamed to the server in real time.

[0813] Input: Voice data during a call

[0814] Output: Real-time audio data sent to the server

[0815] Step 3:

[0816] The server receives the voice data sent from the device in real time and converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[0817] Input: Real-time audio data

[0818] Output: Text data converted by the speech recognition engine

[0819] Step 4:

[0820] The server then analyzes the converted text data using natural language processing (NLP) algorithms (e.g., SpaCy, nltk) to detect specific phrases and contextual patterns that may be indicative of fraud.

[0821] Input: Text data

[0822] Output: Suspicious pattern detection results

[0823] Step 5:

[0824] If a suspicious pattern is detected, the server sends visual and audio alerts to the user's (employee's) device, making them aware of possible fraud and alerting them to the situation.

[0825] Input: Suspicious pattern detection result

[0826] Output: Visual and audio alerts sent to the user's device

[0827] Step 6:

[0828] If the server detects a suspicious pattern, it also sends a notification to the device of a pre-registered administrator, stating that the call content may contain suspicious content.

[0829] Input: Suspicious pattern detection result

[0830] Output: Notification sent to administrator's terminal

[0831] Step 7:

[0832] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent.

[0833] Input: Text data from calls, suspicious pattern detection results, history of alerts and notifications sent

[0834] Output: Call summary report

[0835] Step 8:

[0836] The generated summary report is stored in the server's user database, where users (employees) and managers can review the report at a later time.

[0837] Input: Call Summary Report

[0838] Output: Saved summary report

[0839] Step 9:

[0840] Users (employees) can add new phone numbers to the blacklist, delete them, or edit them. The edited blacklist is sent from the terminal to the server and stored in the database.

[0841] Input: Blacklist edit information

[0842] Output: Updated blacklist

[0843] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0844] The present invention combines an emotion engine with a special fraud prevention system targeted at elderly people, and functions with the cooperation of a terminal, a server, and a user. Specific embodiments of the present invention will be described below.

[0845] 1. Initial device setup

[0846] A special fraud prevention application is installed on the user's device. This application has the function of transmitting the voice during a call to a server in real time and the function of recognizing the user's emotions using an emotion engine. When the user starts the application for the first time, they register the contact information of their family or guardian and perform the initial settings for the blacklist.

[0847] 2. Processing at the start of a call

[0848] Audio capture and transmission

[0849] The device starts capturing the call as soon as it is initiated, and the audio data is then appropriately serialized and streamed to the server in real time.

[0850] 3. Real-time voice analysis and emotion recognition on the server

[0851] Receiving voice data and recognizing voice

[0852] The server receives the voice data sent from the terminal and converts the received voice data into text data using a voice recognition engine.

[0853] Natural language processing and suspicious pattern detection

[0854] The converted text data is then analyzed using natural language processing (NLP) algorithms, which detect specific phrases and contextual patterns that may be indicative of fraud. If a suspicious pattern is detected, a warning flag is raised.

[0855] Emotion recognition by emotion engine

[0856] In parallel, the voice data is fed into an emotion engine to recognize the user's emotions. The emotion engine analyzes voice characteristics such as tone, pitch, and speed to identify the user's emotional state.

[0857] 4. Warnings and Notifications

[0858] Alert the user

[0859] When a warning flag is raised, the server sends visual and audio alerts to the device, whose content and format are adaptively changed based on the user's emotions as recognized by the emotion engine.

[0860] Examples:

[0861] If a suspicious pattern is detected and the user expresses anxiety or fear, a highlighted alert will be displayed stating, "Urgent attention required. Possible scam."

[0862] Notification to family and guardians

[0863] In addition, the server sends a notification to the device of a pre-registered family member or guardian, informing them that the user's call may contain suspicious content and including the user's emotional information as recognized by the emotion engine.

[0864] Examples:

[0865] A message will be sent to the family member's smartphone saying, "A user has made a potentially fraudulent call and we are very concerned. Please check immediately."

[0866] 5. Post-call processing

[0867] Generate summaries of call data

[0868] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, the perceived emotional state, and a history of any alerts or notifications sent.

[0869] Call data storage and management

[0870] The generated reports are stored in the user database so that users and their families can review them later.

[0871] 6. Blacklist Management

[0872] Viewing and Editing the Blacklist

[0873] Users can view, add, remove, or edit the blacklisted phone numbers through the app's interface, which is then sent to the server and stored in a database.

[0874] Examples:

[0875] Users can add newly received phone numbers they suspect to be fraudulent to a blacklist, and the changes are immediately reflected on the server.

[0876] 7. Data Security and Privacy

[0877] Data encryption and protection

[0878] All data sent between the server and the device is protected using a powerful encryption algorithm, reducing the risk of personal information and call content being leaked to third parties.

[0879] User-controlled data

[0880] Users can manage their own call history and data usage settings within the app, allowing them to use the system while ensuring their privacy.

[0881] As described above, this invention can significantly reduce the risk of seniors becoming victims of fraud by integrating real-time call content analysis, a warning system, and emotion recognition. It also provides a user-friendly interface for easy use.

[0882] The processing flow will be explained below.

[0883] Processing at the start of a call

[0884] Step 1: Capture the call audio (on your device)

[0885] 1. Detect the start of a call.

[0886] 2. The system will begin the process of capturing the audio during the call.

[0887] 3. Convert the captured audio data into an appropriate format (e.g., buffer).

[0888] Step 2: Sending audio data (terminal)

[0889] 1. Serialize the converted audio data.

[0890] 2. Stream audio data to the server in real time.

[0891] Real-time analysis of voice data and emotion recognition

[0892] Step 3: Receiving audio data (server)

[0893] 1. Receive the audio data sent from the device.

[0894] 2. Deserialize the received audio data into the appropriate format.

[0895] Step 4: Speech Recognition (Server)

[0896] 1. Pass the deserialized voice data to the speech recognition engine.

[0897] 2. Convert the audio data into text data.

[0898] Step 5: Analysis using natural language processing (NLP) algorithms (server)

[0899] 1. The converted text data is fed into the NLP algorithm.

[0900] 2. Analyze the text using NLP algorithms to detect suspicious patterns (indicators of fraud).

[0901] 3. Raise a warning flag if a suspicious pattern is detected.

[0902] Step 6: Emotion recognition by the emotion engine (server)

[0903] 1. Input the voice data into the emotion engine.

[0904] 2. The emotion engine analyzes the tone, pitch, and rate of speech to identify the user's emotional state.

[0905] 3. The emotional state is recognized and recorded in a database.

[0906] Handling notifications and alerts

[0907] Step 7: Sending alerts to users (server)

[0908] 1. If a warning flag is raised, the server sends a visual and audio alert to the user terminal.

[0909] 2. The content and format of the alerts are adaptively changed based on the user's emotions recognized by the emotion engine.

[0910] Examples:

[0911] If anxiety or fear is detected, a highlighted alert will appear saying, "Urgent attention needed. Possible scam."

[0912] Step 8: Sending notifications to family members and guardians (server)

[0913] 1. If a warning flag is raised, a notification will be sent to the device of a pre-registered family member or guardian.

[0914] 2. The notification will include information about potentially suspicious content in the user's call and the perceived emotional information.

[0915] Examples:

[0916] A message will be sent to family members' smartphones saying, "We are very concerned that a user has made a potentially fraudulent call. Please check immediately."

[0917] Post-call processing

[0918] Step 9: Generate summary of call data (server)

[0919] 1. Once the call is complete, generate a call summary report.

[0920] 2. The report should include a summary of the call, any suspicious patterns detected, any emotional states recognized, and a history of notifications sent.

[0921] Step 10: Storing call data (server)

[0922] 1. The generated call summary is saved in the user database.

[0923] 2. The saved data will be available for review by the user or family at a later date.

[0924] Blacklist Management

[0925] Step 11: View and Edit Blacklist (User)

[0926] 1. The user accesses the app's blacklist management screen.

[0927] 2. View the current blacklist.

[0928] 3. User can add new phone numbers, delete existing numbers and perform edit operations.

[0929] Step 12: Update the Blacklist (Server)

[0930] 1. When a user edits the blacklist, the updates are sent to the server.

[0931] 2. The server saves the new blacklist contents in the database.

[0932] 3. The updated blacklist will be used the next time you receive a call.

[0933] Data Security and Privacy

[0934] Step 13: Encrypt and Secure Your Data (Server)

[0935] 1. Protect data sent between the server and the device with a strong encryption algorithm.

[0936] 2. The encryption technology used prevents third parties from eavesdropping or tampering with the data.

[0937] Step 14: User-managed data (User)

[0938] 1. Users can manage their call history and data usage settings within the app.

[0939] 2. You can take actions to ensure privacy, such as deleting data and changing settings.

[0940] Example 2

[0941] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0942] Special frauds targeting the elderly have been increasing in recent years, making countermeasures urgently needed. The elderly are particularly susceptible to the sophisticated tactics of fraudsters, and the damage they cause affects their families and society as a whole. Furthermore, existing fraud prevention systems lack the accuracy to detect suspicious patterns and have difficulty responding in real time. Furthermore, they lack appropriate alert functions that take into account the user's emotional state, which can prevent the elderly from responding appropriately in emergencies.

[0943] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving voice data transmitted from a terminal in real time and converting the voice data into text data using a voice recognition module; means for analyzing the converted text data using a natural language analysis algorithm to detect suspicious patterns; means for identifying the user's emotional state using an emotion analysis engine when a suspicious pattern is detected; means for sending adaptive visual and audio alerts to the terminal based on the user's emotional state; and means for sending notifications to pre-registered contacts when a suspicious pattern is detected. This significantly reduces the risk of elderly people becoming victims of special fraud and enables rapid response in real time. Furthermore, alerts that take emotional state into account allow users to take appropriate action according to the situation.

[0944] "Audio data" is data that represents an audio signal transmitted from a terminal in digital form.

[0945] A "voice recognition module" is software or hardware for converting voice data into text data.

[0946] "Text data" refers to data that includes character information converted by a voice recognition module.

[0947] A "natural language analysis algorithm" is a set of techniques or processes that analyze text data to detect specific words, phrases, and contexts.

[0948] A "suspicious pattern" is a pattern of specific words, context, or behavior that suggests fraud or wrongdoing.

[0949] An "emotion analysis engine" is a system that recognizes and identifies a user's emotional state from voice and text.

[0950] A "visual alert" is a visual display of a warning message or reminder that is displayed on a device's display.

[0951] A "voice alert" is a warning message or warning alert issued by the device.

[0952] "Registered Contacts" refers to information about family members, guardians, and other emergency contacts that the user has previously registered in the system.

[0953] "Real-time" means that processing and communication occur almost simultaneously with real time.

[0954] "Terminal" refers to a communication device or computer system used by a user, and is a device that constitutes part of this system.

[0955] "Server" refers to a central computer system that receives and processes data from terminals.

[0956] The present invention combines an emotion analysis function with a special fraud prevention system targeted at elderly people, and is a system that functions with the cooperation of terminals, servers, and users. Specific embodiments of the present invention are described in detail below.

[0957] A special fraud prevention application is installed on the user's device. This application has built-in functions for transmitting voice to a server in real time and for recognizing the user's emotions using an emotion analysis engine. When the application is first launched, the user registers the contact information of family members or guardians and sets up a blacklist.

[0958] 1. Initial device setup

[0959] Users install a special fraud prevention application on their device and enter the contact information of their family members or guardians and their blacklist within the application. Once registration is complete, this data is sent to a server and stored in a database.

[0960] 2. When the call starts

[0961] When a user initiates a call, the device automatically starts capturing the call. During the call, audio data is sent to the server in real time, where it is properly serialized and streamed using WebSockets.

[0962] 3. Real-time analysis by the server

[0963] The server receives the voice data sent from the device in real time and converts it into text data using the Google Speech-to-Text API. The converted text data is analyzed using a natural language analysis algorithm (e.g., SpaCy) to detect suspicious patterns. If a suspicious pattern is detected, a warning flag is raised. In parallel, the voice data is sent to an emotion analysis engine to identify the user's emotional state.

[0964] For example, if a user makes a call saying, "Please transfer money," the natural language analysis algorithm will detect potentially fraudulent keywords such as "money" and "transfer" and raise a warning flag.

[0965] 4. Sending alerts and notifications

[0966] When the server receives a warning flag, it sends a visual and audio alert to the user's device. The content of the alert changes adaptively based on the user's emotions, as recognized by an emotion analysis engine. For example, if the user expresses anxiety or fear, a highlighted alert will be displayed stating, "Urgent attention required. Possible fraud." The server also sends notifications to pre-registered contacts to inform the user of a suspicious call.

[0967] For example, a message may be sent to a family member's smartphone saying, "We are very concerned that a user has made a potentially fraudulent call. Please check immediately."

[0968] 5. Post-call processing

[0969] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, the perceived emotional state, and a history of any alerts or notifications sent. The report is stored in the user database for later review by the user or family members.

[0970] 6. Blacklist Management

[0971] Through the application interface, users can view, add, delete and edit the blacklist, and these changes are sent to the server and stored in a database.

[0972] For example, when a user adds a newly received phone number that they believe to be fraudulent to the blacklist, the changes are immediately reflected on the server.

[0973] 7. Data Security and Privacy

[0974] Data transmitted between the server and the device is protected using strong encryption algorithms such as AES-256. In addition, users can manage their call history and data usage settings within the app, ensuring their privacy and allowing them to use the system with peace of mind.

[0975] The system significantly reduces the risk of seniors becoming victims of fraud and enables real-time responses, with adaptive alerts based on emotional state allowing users to take appropriate action quickly.

[0976] Example prompt sentence:

[0977] "I'm thinking about a system to prevent special frauds against the elderly. Could you please give me an overview of the system that analyzes phone calls in real time and issues a warning if there are signs of fraud?"

[0978] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0979] Step 1:

[0980] The user installs the special fraud prevention application on their device. When they first launch it, they enter the contact information and blacklist of their family members or guardians to register them in the application.

[0981] Specifically, the user downloads and installs the application. After launching the application, they enter the necessary contact and blacklist information on the registration screen that appears, and then press the "Save" button. This information is sent to the server and saved in a database.

[0982] Input: User contact information, blacklist information

[0983] Output: Preferences information stored in the server database

[0984] Step 2:

[0985] When a user initiates a call, the device automatically prepares to capture the call content.

[0986] Specifically, the app uses the mobile phone's API to detect when a new call starts, and the moment the call starts, the app automatically launches in the background.

[0987] Input: Call Start Event

[0988] Output: Ready to capture conversation

[0989] Step 3:

[0990] The device captures the audio data of the call and streams it in real time to the server, where it is appropriately serialized and transmitted using WebSockets.

[0991] Specifically, the device stores the call audio in a buffer. Whenever the data in this buffer reaches a certain amount, the audio data is streamed via WebSocket.

[0992] Input: Call audio data

[0993] Output: Serialized audio data sent to the server

[0994] Step 4:

[0995] The server receives the voice data transmitted from the terminal in real time and converts the voice data into text data using a voice recognition module.

[0996] Specifically, when the server receives the voice data, it calls the Google Speech-to-Text API to perform speech recognition, and the results are returned as text data.

[0997] Input: Serialized audio data

[0998] Output: Text data

[0999] Step 5:

[1000] The server then analyzes the converted text data using natural language analysis algorithms to detect suspicious patterns, while a sentiment analysis engine identifies the user's emotional state.

[1001] Specifically, the server analyzes the text data using SpaCy to detect suspicious patterns based on specific keywords and context, while simultaneously sending the audio data to an emotion analysis engine to determine the user's emotional state based on their tone and pitch.

[1002] Input: Text data, audio data

[1003] Output: Presence or absence of suspicious patterns, emotional state

[1004] Step 6:

[1005] If a suspicious pattern is detected, the server sends adaptive visual and audio alerts to the user's device based on their emotional state.

[1006] Specifically, the server checks the warning flag, and if the user shows signs of anxiety or fear, it sends an alert message to the user's device stating, "Urgent attention required. Possible fraud."

[1007] Input: Suspicious pattern detection results, emotional state

[1008] Output: An adaptive alert message sent to the user's device.

[1009] Step 7:

[1010] If a suspicious pattern is detected, the server will also send a notification to pre-registered contacts.

[1011] Specifically, the server will notify the user's family or guardian with a message stating, "The user has made a potentially fraudulent call and is very concerned. Please check immediately."

[1012] Input: Suspicious pattern detection results, emotional state

[1013] Output: Notification message sent to registered contacts

[1014] Step 8:

[1015] Once the call is over, the server generates a summary report of the call and stores it in the user database, including a summary of the call, any suspicious patterns detected, emotional state, and a history of any alerts or notifications sent.

[1016] Specifically, the server generates a summary report based on the full text of the call and emotion recognition data and stores it in a database.

[1017] Input: Full text data of calls, emotion recognition data

[1018] Output: Generated summary report

[1019] Step 9:

[1020] Users can view, add, delete and edit their blacklists, and these changes are sent to the server and stored in the database.

[1021] Specifically, when a user accesses the "Blacklist Management" menu in the app, enters a new phone number, and presses the "Add" button, the number is added to the blacklist. This information is sent to the server and stored in a database.

[1022] Input: User-modified blacklist

[1023] Output: Blacklist information updated in the server database

[1024] (Application example 2)

[1025] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[1026] Seniors need protection from fraud and suspicious transactions when shopping in brick-and-mortar stores. However, current systems lack real-time monitoring and emotion recognition in brick-and-mortar stores, putting seniors at high risk of fraud. Furthermore, existing prevention systems lack adaptive warnings that take into account the user's emotional state, preventing appropriate responses in emergencies.

[1027] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1028] In this invention, the server includes means for receiving voice data transmitted from the terminal in real time and converting the voice data into text data using a voice recognition engine, means for analyzing the converted text data using a natural language processing algorithm to detect suspicious patterns, means for sending visual and audio alerts to the user's terminal when a suspicious pattern is detected, means for sending notifications to terminals of pre-registered family members or guardians when a suspicious pattern is detected, means for analyzing the voice data using an emotion recognition engine to identify the user's emotional state, and means for adaptively changing the corresponding alert format when the user expresses anxiety. This allows elderly people to enjoy shopping in physical stores without worrying about fraud or suspicious transactions.

[1029] A "terminal" is a device that captures audio and video data and transmits it to a server in real time.

[1030] "Voice data" refers to data that records voice information such as calls and conversations in digital format.

[1031] "Real-time" is a term that refers to data being processed immediately as it is acquired, without delay.

[1032] A "voice recognition engine" is a general term for software and algorithms that analyze voice data and convert its contents into text data.

[1033] "Text data" refers to character information converted from voice data by a voice recognition engine.

[1034] A "natural language processing algorithm" is a set of computational methods that allow a computer to process human language and analyze its meaning.

[1035] A "suspicious pattern" refers to a combination of specific words, phrases, or context that may indicate fraud or misconduct.

[1036] "Visual and audio alerts" means visual indications and audio notifications intended to alert the user.

[1037] An "emotion recognition engine" is a technology that uses audio and video data to analyze a user's emotional state.

[1038] "Adaptively changing" means automatically adjusting the system's output and behavior according to the situation or conditions.

[1039] "Family or guardian" refers to a person who supports the user and whose contact information has been registered in the system in advance.

[1040] "Alert Format" means a particular method or means of notifying a user of an urgent event.

[1041] The present invention combines an emotion recognition engine with a special fraud prevention system targeted at elderly people. Specific embodiments are described below.

[1042] 1. System Configuration

[1043] Hardware used

[1044] Smartphone

[1045] Smart Glasses

[1046] Software used

[1047] Speech recognition engine: Google Cloud Speech-to-Text API

[1048] Natural Language Processing Algorithm: BERT Model

[1049] Emotion Recognition Engine: Affectiva API

[1050] Communication protocol: WebRTC

[1051] Alerting system: Firebase Cloud Messaging (FCM)

[1052] 2. Specific implementation method of the system

[1053] User device initial settings

[1054] A fraud prevention application is installed on the user's smartphone or smart glasses. This application has the ability to capture audio and video data in real time and send it to a server. When the application is launched for the first time, the user registers the contact information of family members or guardians and performs initial settings for the blacklist.

[1055] Initiating calls and conversations

[1056] As soon as a call or in-store interaction is initiated, the device begins capturing audio and video data, which is serialized in real time and streamed to the server.

[1057] Data analysis on the server

[1058] The server receives the data sent from the device. The voice data is converted to text data using the Google Cloud Speech-to-Text API. The converted text data is analyzed using a natural language processing algorithm using the BERT model to detect suspicious patterns. If a suspicious pattern is detected, a warning flag is raised.

[1059] In parallel, the voice data is fed into an emotion recognition engine using the Affectiva API to analyze the user's emotional state. Based on the detailed analysis, if the user is expressing anxiety or fear, the alert format will be adaptively changed.

[1060] Alerts and Notifications

[1061] If a suspicious pattern is detected, the server will send a visual and audio alert to the device, such as an alert saying "Urgent attention required. Possible fraud."

[1062] Additionally, notifications will be sent to pre-registered family members and guardians on their devices, stating, "A user has made a potentially fraudulent call, and we are very concerned. Please check immediately."

[1063] Post-call processing

[1064] Once the call or interaction is over, the server generates a summary report of the call and stores it in a database for later review by the user or family members, including a summary of the call, any suspicious pattern detections, emotional state, and a history of any alerts or notifications sent.

[1065] Managing the Blacklist

[1066] Users can view, add, remove, and edit the blacklisted phone numbers and contacts through the app's interface, which is then sent to the server and stored in a database.

[1067] 3. Examples and prompts

[1068] Specific examples

[1069] If a scam indicator is detected in a physical store, such as "You need to give us your credit card details right away" or "You need to complete this process now to get a big discount!"

[1070] The user's smart glasses will play a message saying "Urgent attention required. Possible scam" along with an alert sound.

[1071] A notification will be sent to the family member's smartphone saying, "A user is engaged in a potentially fraudulent transaction and we are very concerned. Please check with the store immediately."

[1072] Prompt Sentence Examples

[1073] Input data: Audio data and real-time streaming

[1074] Emotion recognition API: Affectiva

[1075] Natural Language Processing API: Google Cloud Speech-to-Text, BERT model

[1076] process:

[1077] Converts voice data into text and analyzes it for signs of fraud

[1078] Analyze the user's emotional state

[1079] Triggers warnings when it detects suspicious patterns and user concerns

[1080] output:

[1081] Visual and audio alerts on the user's device

[1082] Notifications on family and guardian devices

[1083] As described above, the present invention provides a system that protects elderly people from fraud and suspicious transactions in physical stores, allowing them to enjoy shopping with peace of mind.By recognizing user emotions and providing adaptive warnings, the system can respond quickly and appropriately in emergencies.

[1084] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1085] Step 1:

[1086] When a call or conversation in a physical store is initiated, the device captures audio and video data using its microphone and camera. This data is streamed to a server in real time using the WebRTC protocol. The input is audio and video data, and the output is serialized streaming data.

[1087] Step 2:

[1088] The server receives the voice data sent from the device in real time. The received voice data is input into the Google Cloud Speech-to-Text API, which converts the voice into text data. The input is voice data, and the output is voice transcription data (text data).

[1089] Step 3:

[1090] The server analyzes text data using a natural language processing algorithm based on the BERT model to detect suspicious patterns. The input is text data, and the output is a flag indicating whether a suspicious pattern exists. Specifically, the server performs grammatical analysis of the text data to detect suspicious words and phrases.

[1091] Step 4:

[1092] The server simultaneously analyzes the voice data with an emotion recognition engine using the Affectiva API to identify the user's emotional state. The input is the voice data, and the output is a classification of the user's emotional state (e.g., anxiety, fear, relief, etc.). Specific operations include analyzing the tone, pitch, and speed of the voice.

[1093] Step 5:

[1094] The server sends visual and audio alerts to the device if a suspicious pattern is detected and the user's emotional state indicates anxiety or fear. The input is a suspicious pattern flag and a classification of the emotional state, and the output is a visual and audio alert. Specific behaviors include adaptively changing the content and format of the alert.

[1095] Step 6:

[1096] If a suspicious pattern is detected, the server also sends a notification to the devices of pre-registered family members or guardians. The input is a suspicious pattern flag and the user's emotional state, and the output is a notification message to the family or guardian. Specifically, the notification is sent using Firebase Cloud Messaging (FCM).

[1097] Step 7:

[1098] Once the call or conversation is over, the server generates a summary report of the call and stores it in a database for later review by the user or family. The input is the text data of the call or conversation, the results of suspicious pattern detection, emotional state, and a history of sent alerts and notifications, and the output is a summary report. Specifically, this data is compiled into a single document and stored in the database.

[1099] Step 8:

[1100] Users can use the app's interface to view, add, delete, and edit blacklisted phone numbers and contacts. The input is the user's instructions, and the output is the updated blacklist information. Specifically, after accepting the user's operation, the app sends the updated blacklist data to the server and updates the database.

[1101] The above processing steps enable real-time audio and video data analysis, emotion recognition, suspicious pattern detection, warning systems, and notification systems to work together in an integrated manner, allowing elderly people to enjoy shopping in physical stores with peace of mind.

[1102] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1103] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1104] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[1105] [Third embodiment]

[1106] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[1107] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[1108] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1109] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[1110] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1111] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1112] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1113] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1114] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1115] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1116] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1117] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[1118] The present invention is a special fraud prevention system for elderly people, which functions with the cooperation of a terminal, a server, and a user. Specific embodiments of the present invention will be described below.

[1119] 1. Initial device setup

[1120] A special fraud prevention application is installed on the user's device. This application has the function of transmitting the audio of calls in real time to a server. When the user starts the application for the first time, they register the contact information of their family members or guardians and perform the initial settings for the blacklist.

[1121] 2. Processing at the start of a call

[1122] Audio capture and transmission

[1123] The device starts capturing the call as soon as it is initiated, and the audio data is then appropriately serialized and streamed to the server in real time.

[1124] 3. Real-time audio analysis on the server

[1125] Receiving voice data and recognizing voice

[1126] The server receives the voice data sent from the terminal and converts the received voice data into text data using a voice recognition engine.

[1127] Natural language processing and suspicious pattern detection

[1128] The converted text data is then analyzed using natural language processing (NLP) algorithms, which detect specific phrases and contextual patterns that may be indicative of fraud. If a suspicious pattern is detected, a warning flag is raised.

[1129] 4. Warnings and Notifications

[1130] Alert the user

[1131] If a warning flag is raised, the server will send a visual and audio alert to the device to make the user aware of the potential fraud and alert them to it.

[1132] Examples:

[1133] When a suspicious pattern is detected, a pop-up message will appear saying, "This is a potential fraudulent call. Please be careful," and an audio warning will also be issued.

[1134] Notification to family and guardians

[1135] Additionally, the server sends a notification to pre-registered family members or guardians, stating that the user's call may contain suspicious content.

[1136] Examples:

[1137] A message will be sent to the family member's smartphone saying, "User's call may be fraudulent. Please check immediately."

[1138] 5. Post-call processing

[1139] Generate summaries of call data

[1140] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent.

[1141] Call data storage and management

[1142] The generated reports are stored in the user database so that users and their families can review them later.

[1143] 6. Blacklist Management

[1144] Viewing and Editing the Blacklist

[1145] Users can view, add, remove, or edit the blacklisted phone numbers through the app's interface, which is then sent to the server and stored in a database.

[1146] Examples:

[1147] Users can add newly received phone numbers they suspect to be fraudulent to a blacklist, and the changes are immediately reflected on the server.

[1148] 7. Data Security and Privacy

[1149] Data encryption and protection

[1150] All data sent between the server and the device is protected using a powerful encryption algorithm, reducing the risk of personal information and call content being leaked to third parties.

[1151] User-controlled data

[1152] Users can manage their own call history and data usage settings within the app, allowing them to use the system while ensuring their privacy.

[1153] As described above, the present invention can significantly reduce the risk of seniors becoming victims of fraud by integrating multiple functions, such as real-time call content analysis, a warning system, and blacklist management, while providing a user-friendly interface for easy use.

[1154] The processing flow will be explained below.

[1155] Processing at the start of a call

[1156] Step 1: Capture the call audio (on your device)

[1157] 1. Detect the start of a call.

[1158] 2. The system will begin the process of capturing the audio during the call.

[1159] 3. Convert the captured audio data into an appropriate format (e.g., buffer).

[1160] Step 2: Sending audio data (terminal)

[1161] 1. Serialize the converted audio data.

[1162] 2. Stream audio data to the server in real time.

[1163] Real-time analysis of voice data

[1164] Step 3: Receiving audio data (server)

[1165] 1. Receive the audio data sent from the device.

[1166] 2. Deserialize the received audio data into the appropriate format.

[1167] Step 4: Speech Recognition (Server)

[1168] 1. Pass the deserialized voice data to the speech recognition engine.

[1169] 2. Convert the audio data into text data.

[1170] Step 5: Natural Language Processing (Server)

[1171] 1. The converted text data is fed into a natural language processing (NLP) algorithm.

[1172] 2. Analyze the text using NLP algorithms to detect suspicious patterns (indicators of fraud).

[1173] 3. Raise a warning flag if a suspicious pattern is detected.

[1174] Handling notifications and alerts

[1175] Step 6: Sending alerts to users (server)

[1176] 1. If a warning flag is raised, send a visual and audio alert to the user's device.

[1177] 2. The alert will include the message "Suspicious call detected."

[1178] Step 7: Notify family and guardians (server)

[1179] 1. If a warning flag is raised, a notification will be sent to the smartphones of pre-registered family members or guardians.

[1180] 2. The notification will include a warning that "your call may contain suspicious content."

[1181] Post-call processing

[1182] Step 8: Generate summary of call data (server)

[1183] 1. Once the call is complete, generate a call summary report.

[1184] 2. The report should include a summary of the call, any suspicious patterns detected, and a history of notifications sent.

[1185] Step 9: Storing call data (server)

[1186] 1. The generated call summary is saved in the user database.

[1187] 2. The saved data will be available for review by the user or family at a later date.

[1188] Blacklist Management

[1189] Step 10: View and Edit Blacklist (User)

[1190] 1. The user accesses the app's blacklist management screen.

[1191] 2. View the current blacklist.

[1192] 3. The user performs an edit operation, such as adding a new phone number or deleting an existing number.

[1193] Step 11: Update the Blacklist (Server)

[1194] 1. When a user edits the blacklist, the updates are sent to the server.

[1195] 2. The server saves the new blacklist contents in the database.

[1196] 3. The updated blacklist will be used the next time you receive a call.

[1197] Data Security and Privacy

[1198] Step 12: Encrypt and Secure Your Data (Server)

[1199] 1. Protect data sent between the server and the device with a strong encryption algorithm.

[1200] 2. The encryption technology used prevents third parties from eavesdropping or tampering with the data.

[1201] Step 13: User-managed data (User)

[1202] 1. Users can manage their call history and data usage settings within the app.

[1203] 2. You can take actions to ensure privacy, such as deleting data and changing settings.

[1204] Example 1

[1205] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1206] Special frauds targeting the elderly are on the rise, and traditional prevention systems have struggled to detect fraud in real time or respond quickly. Furthermore, it is important to not only detect and warn of signs of fraud, but also to record the contents of calls so that they can be reviewed later. There is also a need for systems that allow users to easily manage blacklists and ensure data safety and privacy.

[1207] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1208] In this invention, the server includes: means for receiving voice data transmitted from the terminal in real time and converting the voice data into text data using a voice recognition engine; means for analyzing the converted text data using a natural language processing algorithm to detect suspicious patterns; means for sending visual and audio alerts to the user's terminal when a suspicious pattern is detected; means for sending notifications to the terminals of pre-registered family members or guardians when a suspicious pattern is detected; and means for generating a summary report of the call content after the call ends and storing it in a user database. This enables the detection and prevention of fraud against elderly people in real time, and allows the content of the call to be reviewed later, enabling more effective fraud prevention. It also allows users to easily manage their blacklist, ensuring data security and privacy.

[1209] "Terminal" means a device used by a user to access the special fraud prevention system.

[1210] A "server" is a central computer system that receives and processes data sent from terminals.

[1211] "Audio data" refers to audio information of the contents of a call captured from a terminal.

[1212] A "voice recognition engine" is a software or hardware technology that converts voice data into text data.

[1213] "Text data" is character string information converted from voice data by a voice recognition engine.

[1214] A "natural language processing algorithm" is a computer program that analyzes text data and detects specified patterns or intent.

[1215] A "suspicious pattern" is a combination of specific words, phrases, or context that indicates possible fraud.

[1216] "Visual and audio alerts" are screen displays and audio notifications that notify the user of the detection of a suspicious pattern.

[1217] "Family and guardians" are trusted contacts registered by the user who will receive notifications in the event of an emergency.

[1218] "Notifications" are warning messages sent to family members or guardians when suspicious patterns are detected.

[1219] A "summary report" is a document generated after a call ends that includes a summary of the call, any suspicious pattern detections, and a history of alerts and notifications.

[1220] A "blacklist" is a list where users register phone numbers that may be fraudulent.

[1221] "Database" means a structured data storage for managing information collected and stored by a system.

[1222] The present invention is a special fraud prevention system for elderly people. This system works with the cooperation of terminals, a server, and users.

[1223] 1. Initial device setup

[1224] The user installs a special fraud prevention application on their device. This application has the function of capturing audio data while the user is on a call and sending it to a server in real time. After installation, the user launches the application for the first time and registers the contact information of their family or guardian. They also perform initial setup to add phone numbers that may be fraudulent to a blacklist. This completes the basic preparations for fraud prevention.

[1225] 2. Processing at the start of a call

[1226] The device starts capturing the audio data of the call as soon as the user starts the call, then the captured audio data is serialized into protocol buffers (Protobuf) or JSON format and streamed to the server in real time.

[1227] 3. Real-time audio analysis on the server

[1228] The server receives voice data sent from the device in real time. The received voice data is converted into text data using a speech recognition engine such as Google Cloud Speech-to-Text. The text data is then analyzed using natural language processing (NLP) algorithms (e.g., spaCy or BERT). The NLP algorithms detect specific phrases and contextual patterns that may be indicative of fraud, and if suspicious patterns are detected, a warning flag is raised.

[1229] 4. Warnings and Notifications

[1230] If a warning flag is raised, the server sends visual and audio alerts to the device, allowing the user to recognize and be on guard against possible fraud. For example, a pop-up message or audio alert may appear on the device stating, "This call may be fraudulent. Please be careful." The server also sends warning notifications to the devices of family members or guardians who the user has registered in advance. This notification includes the message, "This user's call may be fraudulent. Please check immediately."

[1231] 5. Post-call processing

[1232] Once the call is completed, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent. The report is then stored in the user database for later review by the user or family.

[1233] 6. Blacklist Management

[1234] The application interface allows users to add, delete, and edit the blacklist phone numbers, allowing users to add newly received phone numbers that they believe to be fraudulent to the blacklist. The edited blacklist is immediately sent to the server and stored in the database.

[1235] 7. Data Security and Privacy

[1236] All data is protected using strong encryption algorithms such as AES-256 when transmitted between the server and the device, reducing the risk of personal information and call content being leaked to third parties. Users can also manage their call history and data usage settings within the app, ensuring privacy while using the system.

[1237] Examples of specific examples and prompts

[1238] Examples:

[1239] A special fraud prevention app installed on the device automatically sends the contents of the call to a server.

[1240] The server analyzes the call content in real time and detects suspicious patterns.

[1241] If detected, an alert will be sent to you and your family.

[1242] Generate and save detailed reports after the call ends.

[1243] Example prompt sentence:

[1244] "Please explain the voice analysis system to prevent seniors from falling victim to fraud."

[1245] "Please explain in detail how your system analyzes call content in real time to prevent fraud."

[1246] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1247] Step 1:

[1248] Install the application on your device.

[1249] The user installs a special fraud prevention application on the device. After installation is complete, the application is launched for the first time, and the user registers the contact information of family members or guardians and adds phone numbers that may be used for fraud to a blacklist. These settings are sent from the device to the server and saved in a database.

[1250] Input: Application installations, contact information, blacklist information

[1251] Output: Configuration data sent to the server

[1252] Step 2:

[1253] Capture the contents of the call.

[1254] When a user initiates a call, the device starts capturing the call audio in real time, and the captured audio data is serialized and sent to the server in streaming format.

[1255] Input: Call audio

[1256] Output: Serialized audio data

[1257] Step 3:

[1258] Convert the audio data into text data.

[1259] The server receives the voice data sent from the terminal and converts the voice data into text data using a voice recognition engine (e.g., Google Cloud Speech-to-Text).

[1260] Input: Serialized audio data

[1261] Output: Text data

[1262] Step 4:

[1263] Analyze text data.

[1264] The server then analyzes the converted text data using natural language processing (NLP) algorithms (e.g., spaCy or BERT) to detect specific phrases and contextual patterns and look for signs of fraud.

[1265] Input: Text data

[1266] Output: Detection result (whether or not there is a suspicious pattern)

[1267] Step 5:

[1268] Send a warning to the user.

[1269] If the server detects a suspicious pattern, it will send a visual and audio alert to the device. The user's device will display a pop-up message saying, "This is a potential fraudulent call. Please be careful," and will also play an audio alert.

[1270] Input: Detection result (whether or not there is a suspicious pattern)

[1271] Output: Visual and audio alerts to user terminal

[1272] Step 6:

[1273] Send notifications to family members and guardians.

[1274] If the server detects a suspicious pattern, it sends a notification to the device of a pre-registered family member or guardian saying, "The user's call may be fraudulent. Please check immediately."

[1275] Input: Detection result (whether or not there is a suspicious pattern)

[1276] Output: Notification to family and guardians

[1277] Step 7:

[1278] Generate a summary report of the call.

[1279] After the call is completed, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent.

[1280] Input: Call text data and detection results

[1281] Output: Summary report

[1282] Step 8:

[1283] Manage blacklists.

[1284] The application's interface allows users to add, remove, and edit potentially fraudulent phone numbers to the blacklist, which is then sent to the server and stored in a database.

[1285] Input: User edits blacklist information

[1286] Output: Updated blacklist information

[1287] Step 9:

[1288] Ensure data security and privacy.

[1289] All data sent between the server and the device is protected using strong encryption algorithms such as AES-256, reducing the risk of personal information and call content being leaked to third parties.

[1290] Input: All data sent and received

[1291] Output: Encrypted data

[1292] (Application example 1)

[1293] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1294] Elderly people and store employees are at high risk of fraud, especially phone fraud. Employees are required to detect suspicious patterns during calls and respond quickly, but current methods make it difficult to provide real-time warnings and notifications, preventing effective prevention. Therefore, there is a need for a system that can detect signs of fraud during voice calls in real time, immediately warn employees, and notify managers.

[1295] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1296] In this invention, the server includes means for receiving voice data transmitted from the terminal in real time and converting the voice data into text data using a voice recognition engine, means for analyzing the converted text data using a natural language processing algorithm to detect suspicious patterns, means for sending visual and audio alerts to the user's terminal when a suspicious pattern is detected, means for capturing voice data during calls received by employees of a physical store based on an application example of the technology and transmitting it to the server for analysis, means for sending a notification to an administrator terminal when a suspicious pattern is detected, and means for generating a summary report of the call content after the call is ended and storing it on the server, thereby enabling real-time detection of signs of fraud and emergency response.

[1297] "Terminal" refers to the communication equipment used by the user, which is a device that captures and transmits audio data.

[1298] "Voice Data" refers to digital information converted from sound wave signals captured during a call and transmitted in real time to a server.

[1299] "Real-time" means that processing occurs nearly simultaneously with the actual event, with minimal delay.

[1300] A "speech recognition engine" is a software or hardware system for converting voice data into text data.

[1301] "Text data" refers to character information converted by a voice recognition engine, and represents the content of voice data as a string of characters.

[1302] A "natural language processing algorithm" is a computer program that analyzes text data and understands its grammatical structure and meaning.

[1303] A "suspicious pattern" is a specific pattern of words or context that indicates possible fraud and is defined based on historical data.

[1304] A "visual alert" is a visual warning message that appears on a user's device to draw attention when a suspicious pattern is detected.

[1305] "Audio alert" is an audio warning signal intended to audibly alert the user.

[1306] "Administrator" refers to a person in charge of system operation and management in an organization such as a physical store.

[1307] A "call summary report" is a report generated after a call ends that includes the key points of the call and any suspicious pattern detections.

[1308] A "blacklist" is a list of potentially fraudulent phone numbers maintained by the system.

[1309] The present invention is a special fraud prevention system, and a specific embodiment will be described that allows employees in physical stores to quickly detect and respond to fraud risks while on the phone. This system functions through the cooperation of terminals, a server, and users (employees and managers). The operation of each element and their interactions will be described in detail below.

[1310] 1. Initial device setup

[1311] A special fraud prevention application is installed on the user's (employee's) device. This application has a function to transmit the audio of a call to a server in real time. When the user starts the application for the first time, they register the administrator's contact information and perform the initial settings for the blacklist.

[1312] 2. Processing at the start of a call

[1313] Audio capture and transmission

[1314] The device starts capturing the call as soon as it is initiated, and the audio data is then appropriately serialized and streamed to the server in real time.

[1315] 3. Real-time audio analysis on the server

[1316] Receiving voice data and recognizing voice

[1317] The server receives the voice data sent from the device, and converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1318] Natural language processing and suspicious pattern detection

[1319] The converted text data is then analyzed using natural language processing (NLP) algorithms (e.g., SpaCy, nltk). NLP algorithms detect specific phrases and contextual patterns that may be indicative of fraud. If a suspicious pattern is detected, a warning flag is raised.

[1320] 4. Warnings and Notifications

[1321] Alert the user

[1322] If a warning flag is raised, the server will send a visual and audio alert to the terminal, which will alert the user (employee) to the possibility of fraud and bring it to their attention.

[1323] Examples:

[1324] When a suspicious pattern is detected, a pop-up message will appear saying, "This is a potential fraudulent call. Please be careful," and an audio warning will also be issued.

[1325] Administrator Notification

[1326] In addition, the server sends a notification to the device of a pre-registered administrator, stating that the user's (employee's) call may contain suspicious content.

[1327] Examples:

[1328] A message is sent to the administrator's smartphone saying, "An employee's call may be fraudulent. Please check immediately."

[1329] 5. Post-call processing

[1330] Generate summaries of call data

[1331] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent.

[1332] Call data storage and management

[1333] The generated reports are stored in the user database so that users (employees) and administrators can review them later.

[1334] 6. Blacklist Management

[1335] Viewing and Editing the Blacklist

[1336] Users (employees) can use the app interface to view, add, delete, and edit the blacklisted phone numbers, which are then sent to the server and stored in a database.

[1337] Examples:

[1338] Users (employees) can add newly received phone numbers that they believe to be fraudulent to the blacklist, and the changes are immediately reflected on the server.

[1339] 7. Data Security and Privacy

[1340] Data encryption and protection

[1341] All data sent between the server and the device is protected using a powerful encryption algorithm, reducing the risk of personal information and call content being leaked to third parties.

[1342] User-controlled data

[1343] Users (employees) can manage their own call history and data usage settings within the app, allowing them to use the system while ensuring privacy.

[1344] Prompt Sentence Examples

[1345] "I want to develop a system that detects potential fraud for employees taking customer orders over the phone. This system would analyze the audio during the call in real time and alert employees and managers if it detects signs of fraud. Please write code using Python and PyAudio to capture audio data and send it to a server. The server would use a speech recognition and natural language processing engine. Please also include a notification function if a suspicious pattern is detected."

[1346] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1347] Step 1:

[1348] The user (employee) starts the special fraud prevention application on the terminal and performs the initial setup. This requires registering the administrator's contact information and initial setup of the blacklist. This inputs basic information into the terminal for the system to operate properly.

[1349] Input: Administrator contact information, blacklist information

[1350] Output: Device with initial setup completed

[1351] Step 2:

[1352] When a call is initiated, the device begins capturing audio data through the microphone, which is then serialized and streamed to the server in real time.

[1353] Input: Voice data during a call

[1354] Output: Real-time audio data sent to the server

[1355] Step 3:

[1356] The server receives the voice data sent from the device in real time and converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1357] Input: Real-time audio data

[1358] Output: Text data converted by the speech recognition engine

[1359] Step 4:

[1360] The server then analyzes the converted text data using natural language processing (NLP) algorithms (e.g., SpaCy, nltk) to detect specific phrases and contextual patterns that may be indicative of fraud.

[1361] Input: Text data

[1362] Output: Suspicious pattern detection results

[1363] Step 5:

[1364] If a suspicious pattern is detected, the server sends visual and audio alerts to the user's (employee's) device, making them aware of possible fraud and alerting them to the situation.

[1365] Input: Suspicious pattern detection result

[1366] Output: Visual and audio alerts sent to the user's device

[1367] Step 6:

[1368] If the server detects a suspicious pattern, it also sends a notification to the device of a pre-registered administrator, stating that the call content may contain suspicious content.

[1369] Input: Suspicious pattern detection result

[1370] Output: Notification sent to administrator's terminal

[1371] Step 7:

[1372] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent.

[1373] Input: Text data from calls, suspicious pattern detection results, history of alerts and notifications sent

[1374] Output: Call summary report

[1375] Step 8:

[1376] The generated summary report is stored in the server's user database, where users (employees) and managers can review the report at a later time.

[1377] Input: Call Summary Report

[1378] Output: Saved summary report

[1379] Step 9:

[1380] Users (employees) can add new phone numbers to the blacklist, delete them, or edit them. The edited blacklist is sent from the terminal to the server and stored in the database.

[1381] Input: Blacklist edit information

[1382] Output: Updated blacklist

[1383] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1384] The present invention combines an emotion engine with a special fraud prevention system targeted at elderly people, and functions with the cooperation of a terminal, a server, and a user. Specific embodiments of the present invention will be described below.

[1385] 1. Initial device setup

[1386] A special fraud prevention application is installed on the user's device. This application has the function of transmitting the voice during a call to a server in real time and the function of recognizing the user's emotions using an emotion engine. When the user starts the application for the first time, they register the contact information of their family or guardian and perform the initial settings for the blacklist.

[1387] 2. Processing at the start of a call

[1388] Audio capture and transmission

[1389] The device starts capturing the call as soon as it is initiated, and the audio data is then appropriately serialized and streamed to the server in real time.

[1390] 3. Real-time voice analysis and emotion recognition on the server

[1391] Receiving voice data and recognizing voice

[1392] The server receives the voice data sent from the terminal and converts the received voice data into text data using a voice recognition engine.

[1393] Natural language processing and suspicious pattern detection

[1394] The converted text data is then analyzed using natural language processing (NLP) algorithms, which detect specific phrases and contextual patterns that may be indicative of fraud. If a suspicious pattern is detected, a warning flag is raised.

[1395] Emotion recognition by emotion engine

[1396] In parallel, the voice data is fed into an emotion engine to recognize the user's emotions. The emotion engine analyzes voice characteristics such as tone, pitch, and speed to identify the user's emotional state.

[1397] 4. Warnings and Notifications

[1398] Alert the user

[1399] When a warning flag is raised, the server sends visual and audio alerts to the device, whose content and format are adaptively changed based on the user's emotions as recognized by the emotion engine.

[1400] Examples:

[1401] If a suspicious pattern is detected and the user expresses anxiety or fear, a highlighted alert will be displayed stating, "Urgent attention required. Possible scam."

[1402] Notification to family and guardians

[1403] In addition, the server sends a notification to the device of a pre-registered family member or guardian, informing them that the user's call may contain suspicious content and including the user's emotional information as recognized by the emotion engine.

[1404] Examples:

[1405] A message will be sent to the family member's smartphone saying, "A user has made a potentially fraudulent call and we are very concerned. Please check immediately."

[1406] 5. Post-call processing

[1407] Generate summaries of call data

[1408] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, the perceived emotional state, and a history of any alerts or notifications sent.

[1409] Call data storage and management

[1410] The generated reports are stored in the user database so that users and their families can review them later.

[1411] 6. Blacklist Management

[1412] Viewing and Editing the Blacklist

[1413] Users can view, add, remove, or edit the blacklisted phone numbers through the app's interface, which is then sent to the server and stored in a database.

[1414] Examples:

[1415] Users can add newly received phone numbers they suspect to be fraudulent to a blacklist, and the changes are immediately reflected on the server.

[1416] 7. Data Security and Privacy

[1417] Data encryption and protection

[1418] All data sent between the server and the device is protected using a powerful encryption algorithm, reducing the risk of personal information and call content being leaked to third parties.

[1419] User-controlled data

[1420] Users can manage their own call history and data usage settings within the app, allowing them to use the system while ensuring their privacy.

[1421] As described above, this invention can significantly reduce the risk of seniors becoming victims of fraud by integrating real-time call content analysis, a warning system, and emotion recognition. It also provides a user-friendly interface for easy use.

[1422] The processing flow will be explained below.

[1423] Processing at the start of a call

[1424] Step 1: Capture the call audio (on your device)

[1425] 1. Detect the start of a call.

[1426] 2. The system will begin the process of capturing the audio during the call.

[1427] 3. Convert the captured audio data into an appropriate format (e.g., buffer).

[1428] Step 2: Sending audio data (terminal)

[1429] 1. Serialize the converted audio data.

[1430] 2. Stream audio data to the server in real time.

[1431] Real-time analysis of voice data and emotion recognition

[1432] Step 3: Receiving audio data (server)

[1433] 1. Receive the audio data sent from the device.

[1434] 2. Deserialize the received audio data into the appropriate format.

[1435] Step 4: Speech Recognition (Server)

[1436] 1. Pass the deserialized voice data to the speech recognition engine.

[1437] 2. Convert the audio data into text data.

[1438] Step 5: Analysis using natural language processing (NLP) algorithms (server)

[1439] 1. The converted text data is fed into the NLP algorithm.

[1440] 2. Analyze the text using NLP algorithms to detect suspicious patterns (indicators of fraud).

[1441] 3. Raise a warning flag if a suspicious pattern is detected.

[1442] Step 6: Emotion recognition by the emotion engine (server)

[1443] 1. Input the voice data into the emotion engine.

[1444] 2. The emotion engine analyzes the tone, pitch, and rate of speech to identify the user's emotional state.

[1445] 3. The emotional state is recognized and recorded in a database.

[1446] Handling notifications and alerts

[1447] Step 7: Sending alerts to users (server)

[1448] 1. If a warning flag is raised, the server sends a visual and audio alert to the user terminal.

[1449] 2. The content and format of the alerts are adaptively changed based on the user's emotions recognized by the emotion engine.

[1450] Examples:

[1451] If anxiety or fear is detected, a highlighted alert will appear saying, "Urgent attention needed. Possible scam."

[1452] Step 8: Sending notifications to family members and guardians (server)

[1453] 1. If a warning flag is raised, a notification will be sent to the device of a pre-registered family member or guardian.

[1454] 2. The notification will include information about potentially suspicious content in the user's call and the perceived emotional information.

[1455] Examples:

[1456] A message will be sent to family members' smartphones saying, "We are very concerned that a user has made a potentially fraudulent call. Please check immediately."

[1457] Post-call processing

[1458] Step 9: Generate summary of call data (server)

[1459] 1. Once the call is complete, generate a call summary report.

[1460] 2. The report should include a summary of the call, any suspicious patterns detected, any emotional states recognized, and a history of notifications sent.

[1461] Step 10: Storing call data (server)

[1462] 1. The generated call summary is saved in the user database.

[1463] 2. The saved data will be available for review by the user or family at a later date.

[1464] Blacklist Management

[1465] Step 11: View and Edit Blacklist (User)

[1466] 1. The user accesses the app's blacklist management screen.

[1467] 2. View the current blacklist.

[1468] 3. User can add new phone numbers, delete existing numbers and perform edit operations.

[1469] Step 12: Update the Blacklist (Server)

[1470] 1. When a user edits the blacklist, the updates are sent to the server.

[1471] 2. The server saves the new blacklist contents in the database.

[1472] 3. The updated blacklist will be used the next time you receive a call.

[1473] Data Security and Privacy

[1474] Step 13: Encrypt and Secure Your Data (Server)

[1475] 1. Protect data sent between the server and the device with a strong encryption algorithm.

[1476] 2. The encryption technology used prevents third parties from eavesdropping or tampering with the data.

[1477] Step 14: User-managed data (User)

[1478] 1. Users can manage their call history and data usage settings within the app.

[1479] 2. You can take actions to ensure privacy, such as deleting data and changing settings.

[1480] Example 2

[1481] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1482] Special frauds targeting the elderly have been increasing in recent years, making countermeasures urgently needed. The elderly are particularly susceptible to the sophisticated tactics of fraudsters, and the damage they cause affects their families and society as a whole. Furthermore, existing fraud prevention systems lack the accuracy to detect suspicious patterns and have difficulty responding in real time. Furthermore, they lack appropriate alert functions that take into account the user's emotional state, which can prevent the elderly from responding appropriately in emergencies.

[1483] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving voice data transmitted from a terminal in real time and converting the voice data into text data using a voice recognition module; means for analyzing the converted text data using a natural language analysis algorithm to detect suspicious patterns; means for identifying the user's emotional state using an emotion analysis engine when a suspicious pattern is detected; means for sending adaptive visual and audio alerts to the terminal based on the user's emotional state; and means for sending notifications to pre-registered contacts when a suspicious pattern is detected. This significantly reduces the risk of elderly people becoming victims of special fraud and enables rapid response in real time. Furthermore, alerts that take emotional state into account allow users to take appropriate action according to the situation.

[1484] "Audio data" is data that represents an audio signal transmitted from a terminal in digital form.

[1485] A "voice recognition module" is software or hardware for converting voice data into text data.

[1486] "Text data" refers to data that includes character information converted by a voice recognition module.

[1487] A "natural language analysis algorithm" is a set of techniques or processes that analyze text data to detect specific words, phrases, and contexts.

[1488] A "suspicious pattern" is a pattern of specific words, context, or behavior that suggests fraud or wrongdoing.

[1489] An "emotion analysis engine" is a system that recognizes and identifies a user's emotional state from voice and text.

[1490] A "visual alert" is a visual display of a warning message or reminder that is displayed on a device's display.

[1491] A "voice alert" is a warning message or warning alert issued by the device.

[1492] "Registered Contacts" refers to information about family members, guardians, and other emergency contacts that the user has previously registered in the system.

[1493] "Real-time" means that processing and communication occur almost simultaneously with real time.

[1494] "Terminal" refers to a communication device or computer system used by a user, and is a device that constitutes part of this system.

[1495] "Server" refers to a central computer system that receives and processes data from terminals.

[1496] The present invention combines an emotion analysis function with a special fraud prevention system targeted at elderly people, and is a system that functions with the cooperation of terminals, servers, and users. Specific embodiments of the present invention are described in detail below.

[1497] A special fraud prevention application is installed on the user's device. This application has built-in functions for transmitting voice to a server in real time and for recognizing the user's emotions using an emotion analysis engine. When the application is first launched, the user registers the contact information of family members or guardians and sets up a blacklist.

[1498] 1. Initial device setup

[1499] Users install a special fraud prevention application on their device and enter the contact information of their family members or guardians and their blacklist within the application. Once registration is complete, this data is sent to a server and stored in a database.

[1500] 2. When the call starts

[1501] When a user initiates a call, the device automatically starts capturing the call. During the call, audio data is sent to the server in real time, where it is properly serialized and streamed using WebSockets.

[1502] 3. Real-time analysis by the server

[1503] The server receives the voice data sent from the device in real time and converts it into text data using the Google Speech-to-Text API. The converted text data is analyzed using a natural language analysis algorithm (e.g., SpaCy) to detect suspicious patterns. If a suspicious pattern is detected, a warning flag is raised. In parallel, the voice data is sent to an emotion analysis engine to identify the user's emotional state.

[1504] For example, if a user makes a call saying, "Please transfer money," the natural language analysis algorithm will detect potentially fraudulent keywords such as "money" and "transfer" and raise a warning flag.

[1505] 4. Sending alerts and notifications

[1506] When the server receives a warning flag, it sends a visual and audio alert to the user's device. The content of the alert changes adaptively based on the user's emotions, as recognized by an emotion analysis engine. For example, if the user expresses anxiety or fear, a highlighted alert will be displayed stating, "Urgent attention required. Possible fraud." The server also sends notifications to pre-registered contacts to inform the user of a suspicious call.

[1507] For example, a message may be sent to a family member's smartphone saying, "We are very concerned that a user has made a potentially fraudulent call. Please check immediately."

[1508] 5. Post-call processing

[1509] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, the perceived emotional state, and a history of any alerts or notifications sent. The report is stored in the user database for later review by the user or family members.

[1510] 6. Blacklist Management

[1511] Through the application interface, users can view, add, delete and edit the blacklist, and these changes are sent to the server and stored in a database.

[1512] For example, when a user adds a newly received phone number that they believe to be fraudulent to the blacklist, the changes are immediately reflected on the server.

[1513] 7. Data Security and Privacy

[1514] Data transmitted between the server and the device is protected using strong encryption algorithms such as AES-256. In addition, users can manage their call history and data usage settings within the app, ensuring their privacy and allowing them to use the system with peace of mind.

[1515] The system significantly reduces the risk of seniors becoming victims of fraud and enables real-time responses, with adaptive alerts based on emotional state allowing users to take appropriate action quickly.

[1516] Example prompt sentence:

[1517] "I'm thinking about a system to prevent special frauds against the elderly. Could you please give me an overview of the system that analyzes phone calls in real time and issues a warning if there are signs of fraud?"

[1518] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1519] Step 1:

[1520] The user installs the special fraud prevention application on their device. When they first launch it, they enter the contact information and blacklist of their family members or guardians to register them in the application.

[1521] Specifically, the user downloads and installs the application. After launching the application, they enter the necessary contact and blacklist information on the registration screen that appears, and then press the "Save" button. This information is sent to the server and saved in a database.

[1522] Input: User contact information, blacklist information

[1523] Output: Preferences information stored in the server database

[1524] Step 2:

[1525] When a user initiates a call, the device automatically prepares to capture the call content.

[1526] Specifically, the app uses the mobile phone's API to detect when a new call starts, and the moment the call starts, the app automatically launches in the background.

[1527] Input: Call Start Event

[1528] Output: Ready to capture conversation

[1529] Step 3:

[1530] The device captures the audio data of the call and streams it in real time to the server, where it is appropriately serialized and transmitted using WebSockets.

[1531] Specifically, the device stores the call audio in a buffer. Whenever the data in this buffer reaches a certain amount, the audio data is streamed via WebSocket.

[1532] Input: Call audio data

[1533] Output: Serialized audio data sent to the server

[1534] Step 4:

[1535] The server receives the voice data transmitted from the terminal in real time and converts the voice data into text data using a voice recognition module.

[1536] Specifically, when the server receives the voice data, it calls the Google Speech-to-Text API to perform speech recognition, and the results are returned as text data.

[1537] Input: Serialized audio data

[1538] Output: Text data

[1539] Step 5:

[1540] The server then analyzes the converted text data using natural language analysis algorithms to detect suspicious patterns, while a sentiment analysis engine identifies the user's emotional state.

[1541] Specifically, the server analyzes the text data using SpaCy to detect suspicious patterns based on specific keywords and context, while simultaneously sending the audio data to an emotion analysis engine to determine the user's emotional state based on their tone and pitch.

[1542] Input: Text data, audio data

[1543] Output: Presence or absence of suspicious patterns, emotional state

[1544] Step 6:

[1545] If a suspicious pattern is detected, the server sends adaptive visual and audio alerts to the user's device based on their emotional state.

[1546] Specifically, the server checks the warning flag, and if the user shows signs of anxiety or fear, it sends an alert message to the user's device stating, "Urgent attention required. Possible fraud."

[1547] Input: Suspicious pattern detection results, emotional state

[1548] Output: An adaptive alert message sent to the user's device.

[1549] Step 7:

[1550] If a suspicious pattern is detected, the server will also send a notification to pre-registered contacts.

[1551] Specifically, the server will notify the user's family or guardian with a message stating, "The user has made a potentially fraudulent call and is very concerned. Please check immediately."

[1552] Input: Suspicious pattern detection results, emotional state

[1553] Output: Notification message sent to registered contacts

[1554] Step 8:

[1555] Once the call is over, the server generates a summary report of the call and stores it in the user database, including a summary of the call, any suspicious patterns detected, emotional state, and a history of any alerts or notifications sent.

[1556] Specifically, the server generates a summary report based on the full text of the call and emotion recognition data and stores it in a database.

[1557] Input: Full text data of calls, emotion recognition data

[1558] Output: Generated summary report

[1559] Step 9:

[1560] Users can view, add, delete and edit their blacklists, and these changes are sent to the server and stored in the database.

[1561] Specifically, when a user accesses the "Blacklist Management" menu in the app, enters a new phone number, and presses the "Add" button, the number is added to the blacklist. This information is sent to the server and stored in a database.

[1562] Input: User-modified blacklist

[1563] Output: Blacklist information updated in the server database

[1564] (Application example 2)

[1565] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1566] Seniors need protection from fraud and suspicious transactions when shopping in brick-and-mortar stores. However, current systems lack real-time monitoring and emotion recognition in brick-and-mortar stores, putting seniors at high risk of fraud. Furthermore, existing prevention systems lack adaptive warnings that take into account the user's emotional state, preventing appropriate responses in emergencies.

[1567] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1568] In this invention, the server includes means for receiving voice data transmitted from the terminal in real time and converting the voice data into text data using a voice recognition engine, means for analyzing the converted text data using a natural language processing algorithm to detect suspicious patterns, means for sending visual and audio alerts to the user's terminal when a suspicious pattern is detected, means for sending notifications to terminals of pre-registered family members or guardians when a suspicious pattern is detected, means for analyzing the voice data using an emotion recognition engine to identify the user's emotional state, and means for adaptively changing the corresponding alert format when the user expresses anxiety. This allows elderly people to enjoy shopping in physical stores without worrying about fraud or suspicious transactions.

[1569] A "terminal" is a device that captures audio and video data and transmits it to a server in real time.

[1570] "Voice data" refers to data that records voice information such as calls and conversations in digital format.

[1571] "Real-time" is a term that refers to data being processed immediately as it is acquired, without delay.

[1572] A "voice recognition engine" is a general term for software and algorithms that analyze voice data and convert its contents into text data.

[1573] "Text data" refers to character information converted from voice data by a voice recognition engine.

[1574] A "natural language processing algorithm" is a set of computational methods that allow a computer to process human language and analyze its meaning.

[1575] A "suspicious pattern" refers to a combination of specific words, phrases, or context that may indicate fraud or misconduct.

[1576] "Visual and audio alerts" means visual indications and audio notifications intended to alert the user.

[1577] An "emotion recognition engine" is a technology that uses audio and video data to analyze a user's emotional state.

[1578] "Adaptively changing" means automatically adjusting the system's output and behavior according to the situation or conditions.

[1579] "Family or guardian" refers to a person who supports the user and whose contact information has been registered in the system in advance.

[1580] "Alert Format" means a particular method or means of notifying a user of an urgent event.

[1581] The present invention combines an emotion recognition engine with a special fraud prevention system targeted at elderly people. Specific embodiments are described below.

[1582] 1. System Configuration

[1583] Hardware used

[1584] Smartphone

[1585] Smart Glasses

[1586] Software used

[1587] Speech recognition engine: Google Cloud Speech-to-Text API

[1588] Natural Language Processing Algorithm: BERT Model

[1589] Emotion Recognition Engine: Affectiva API

[1590] Communication protocol: WebRTC

[1591] Alerting system: Firebase Cloud Messaging (FCM)

[1592] 2. Specific implementation method of the system

[1593] User device initial settings

[1594] A fraud prevention application is installed on the user's smartphone or smart glasses. This application has the ability to capture audio and video data in real time and send it to a server. When the application is launched for the first time, the user registers the contact information of family members or guardians and performs initial settings for the blacklist.

[1595] Initiating calls and conversations

[1596] As soon as a call or in-store interaction is initiated, the device begins capturing audio and video data, which is serialized in real time and streamed to the server.

[1597] Data analysis on the server

[1598] The server receives the data sent from the device. The voice data is converted to text data using the Google Cloud Speech-to-Text API. The converted text data is analyzed using a natural language processing algorithm using the BERT model to detect suspicious patterns. If a suspicious pattern is detected, a warning flag is raised.

[1599] In parallel, the voice data is fed into an emotion recognition engine using the Affectiva API to analyze the user's emotional state. Based on the detailed analysis, if the user is expressing anxiety or fear, the alert format will be adaptively changed.

[1600] Alerts and Notifications

[1601] If a suspicious pattern is detected, the server will send a visual and audio alert to the device, such as an alert saying "Urgent attention required. Possible fraud."

[1602] Additionally, notifications will be sent to pre-registered family members and guardians on their devices, stating, "A user has made a potentially fraudulent call, and we are very concerned. Please check immediately."

[1603] Post-call processing

[1604] Once the call or interaction is over, the server generates a summary report of the call and stores it in a database for later review by the user or family members, including a summary of the call, any suspicious pattern detections, emotional state, and a history of any alerts or notifications sent.

[1605] Managing the Blacklist

[1606] Users can view, add, remove, and edit the blacklisted phone numbers and contacts through the app's interface, which is then sent to the server and stored in a database.

[1607] 3. Examples and prompts

[1608] Specific examples

[1609] If a scam indicator is detected in a physical store, such as "You need to give us your credit card details right away" or "You need to complete this process now to get a big discount!"

[1610] The user's smart glasses will play a message saying "Urgent attention required. Possible scam" along with an alert sound.

[1611] A notification will be sent to the family member's smartphone saying, "A user is engaged in a potentially fraudulent transaction and we are very concerned. Please check with the store immediately."

[1612] Prompt Sentence Examples

[1613] Input data: Audio data and real-time streaming

[1614] Emotion recognition API: Affectiva

[1615] Natural Language Processing API: Google Cloud Speech-to-Text, BERT model

[1616] process:

[1617] Converts voice data into text and analyzes it for signs of fraud

[1618] Analyze the user's emotional state

[1619] Triggers warnings when it detects suspicious patterns and user concerns

[1620] output:

[1621] Visual and audio alerts on the user's device

[1622] Notifications on family and guardian devices

[1623] As described above, the present invention provides a system that protects elderly people from fraud and suspicious transactions in physical stores, allowing them to enjoy shopping with peace of mind.By recognizing user emotions and providing adaptive warnings, the system can respond quickly and appropriately in emergencies.

[1624] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1625] Step 1:

[1626] When a call or conversation in a physical store is initiated, the device captures audio and video data using its microphone and camera. This data is streamed to a server in real time using the WebRTC protocol. The input is audio and video data, and the output is serialized streaming data.

[1627] Step 2:

[1628] The server receives the voice data sent from the device in real time. The received voice data is input into the Google Cloud Speech-to-Text API, which converts the voice into text data. The input is voice data, and the output is voice transcription data (text data).

[1629] Step 3:

[1630] The server analyzes text data using a natural language processing algorithm based on the BERT model to detect suspicious patterns. The input is text data, and the output is a flag indicating whether a suspicious pattern exists. Specifically, the server performs grammatical analysis of the text data to detect suspicious words and phrases.

[1631] Step 4:

[1632] The server simultaneously analyzes the voice data with an emotion recognition engine using the Affectiva API to identify the user's emotional state. The input is the voice data, and the output is a classification of the user's emotional state (e.g., anxiety, fear, relief, etc.). Specific operations include analyzing the tone, pitch, and speed of the voice.

[1633] Step 5:

[1634] The server sends visual and audio alerts to the device if a suspicious pattern is detected and the user's emotional state indicates anxiety or fear. The input is a suspicious pattern flag and a classification of the emotional state, and the output is a visual and audio alert. Specific behaviors include adaptively changing the content and format of the alert.

[1635] Step 6:

[1636] If a suspicious pattern is detected, the server also sends a notification to the devices of pre-registered family members or guardians. The input is a suspicious pattern flag and the user's emotional state, and the output is a notification message to the family or guardian. Specifically, the notification is sent using Firebase Cloud Messaging (FCM).

[1637] Step 7:

[1638] Once the call or conversation is over, the server generates a summary report of the call and stores it in a database for later review by the user or family. The input is the text data of the call or conversation, the results of suspicious pattern detection, emotional state, and a history of sent alerts and notifications, and the output is a summary report. Specifically, this data is compiled into a single document and stored in the database.

[1639] Step 8:

[1640] Users can use the app's interface to view, add, delete, and edit blacklisted phone numbers and contacts. The input is the user's instructions, and the output is the updated blacklist information. Specifically, after accepting the user's operation, the app sends the updated blacklist data to the server and updates the database.

[1641] The above processing steps enable real-time audio and video data analysis, emotion recognition, suspicious pattern detection, warning systems, and notification systems to work together in an integrated manner, allowing elderly people to enjoy shopping in physical stores with peace of mind.

[1642] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1643] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1644] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1645] [Fourth embodiment]

[1646] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1647] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1648] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1649] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1650] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1651] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1652] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1653] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1654] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1655] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1656] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1657] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1658] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1659] The present invention is a special fraud prevention system for elderly people, which functions with the cooperation of a terminal, a server, and a user. Specific embodiments of the present invention will be described below.

[1660] 1. Initial device setup

[1661] A special fraud prevention application is installed on the user's device. This application has the function of transmitting the audio of calls in real time to a server. When the user starts the application for the first time, they register the contact information of their family members or guardians and perform the initial settings for the blacklist.

[1662] 2. Processing at the start of a call

[1663] Audio capture and transmission

[1664] The device starts capturing the call as soon as it is initiated, and the audio data is then appropriately serialized and streamed to the server in real time.

[1665] 3. Real-time audio analysis on the server

[1666] Receiving voice data and recognizing voice

[1667] The server receives the voice data sent from the terminal and converts the received voice data into text data using a voice recognition engine.

[1668] Natural language processing and suspicious pattern detection

[1669] The converted text data is then analyzed using natural language processing (NLP) algorithms, which detect specific phrases and contextual patterns that may be indicative of fraud. If a suspicious pattern is detected, a warning flag is raised.

[1670] 4. Warnings and Notifications

[1671] Alert the user

[1672] If a warning flag is raised, the server will send a visual and audio alert to the device to make the user aware of the potential fraud and alert them to it.

[1673] Examples:

[1674] When a suspicious pattern is detected, a pop-up message will appear saying, "This is a potential fraudulent call. Please be careful," and an audio warning will also be issued.

[1675] Notification to family and guardians

[1676] Additionally, the server sends a notification to pre-registered family members or guardians, stating that the user's call may contain suspicious content.

[1677] Examples:

[1678] A message will be sent to the family member's smartphone saying, "User's call may be fraudulent. Please check immediately."

[1679] 5. Post-call processing

[1680] Generate summaries of call data

[1681] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent.

[1682] Call data storage and management

[1683] The generated reports are stored in the user database so that users and their families can review them later.

[1684] 6. Blacklist Management

[1685] Viewing and Editing the Blacklist

[1686] Users can view, add, remove, or edit the blacklisted phone numbers through the app's interface, which is then sent to the server and stored in a database.

[1687] Examples:

[1688] Users can add newly received phone numbers they suspect to be fraudulent to a blacklist, and the changes are immediately reflected on the server.

[1689] 7. Data Security and Privacy

[1690] Data encryption and protection

[1691] All data sent between the server and the device is protected using a powerful encryption algorithm, reducing the risk of personal information and call content being leaked to third parties.

[1692] User-controlled data

[1693] Users can manage their own call history and data usage settings within the app, allowing them to use the system while ensuring their privacy.

[1694] As described above, the present invention can significantly reduce the risk of seniors becoming victims of fraud by integrating multiple functions, such as real-time call content analysis, a warning system, and blacklist management, while providing a user-friendly interface for easy use.

[1695] The processing flow will be explained below.

[1696] Processing at the start of a call

[1697] Step 1: Capture the call audio (on your device)

[1698] 1. Detect the start of a call.

[1699] 2. The system will begin the process of capturing the audio during the call.

[1700] 3. Convert the captured audio data into an appropriate format (e.g., buffer).

[1701] Step 2: Sending audio data (terminal)

[1702] 1. Serialize the converted audio data.

[1703] 2. Stream audio data to the server in real time.

[1704] Real-time analysis of voice data

[1705] Step 3: Receiving audio data (server)

[1706] 1. Receive the audio data sent from the device.

[1707] 2. Deserialize the received audio data into the appropriate format.

[1708] Step 4: Speech Recognition (Server)

[1709] 1. Pass the deserialized voice data to the speech recognition engine.

[1710] 2. Convert the audio data into text data.

[1711] Step 5: Natural Language Processing (Server)

[1712] 1. The converted text data is fed into a natural language processing (NLP) algorithm.

[1713] 2. Analyze the text using NLP algorithms to detect suspicious patterns (indicators of fraud).

[1714] 3. Raise a warning flag if a suspicious pattern is detected.

[1715] Handling notifications and alerts

[1716] Step 6: Sending alerts to users (server)

[1717] 1. If a warning flag is raised, send a visual and audio alert to the user's device.

[1718] 2. The alert will include the message "Suspicious call detected."

[1719] Step 7: Notify family and guardians (server)

[1720] 1. If a warning flag is raised, a notification will be sent to the smartphones of pre-registered family members or guardians.

[1721] 2. The notification will include a warning that "your call may contain suspicious content."

[1722] Post-call processing

[1723] Step 8: Generate summary of call data (server)

[1724] 1. Once the call is complete, generate a call summary report.

[1725] 2. The report should include a summary of the call, any suspicious patterns detected, and a history of notifications sent.

[1726] Step 9: Storing call data (server)

[1727] 1. The generated call summary is saved in the user database.

[1728] 2. The saved data will be available for review by the user or family at a later date.

[1729] Blacklist Management

[1730] Step 10: View and Edit Blacklist (User)

[1731] 1. The user accesses the app's blacklist management screen.

[1732] 2. View the current blacklist.

[1733] 3. The user performs an edit operation, such as adding a new phone number or deleting an existing number.

[1734] Step 11: Update the Blacklist (Server)

[1735] 1. When a user edits the blacklist, the updates are sent to the server.

[1736] 2. The server saves the new blacklist contents in the database.

[1737] 3. The updated blacklist will be used the next time you receive a call.

[1738] Data Security and Privacy

[1739] Step 12: Encrypt and Secure Your Data (Server)

[1740] 1. Protect data sent between the server and the device with a strong encryption algorithm.

[1741] 2. The encryption technology used prevents third parties from eavesdropping or tampering with the data.

[1742] Step 13: User-managed data (User)

[1743] 1. Users can manage their call history and data usage settings within the app.

[1744] 2. You can take actions to ensure privacy, such as deleting data and changing settings.

[1745] Example 1

[1746] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1747] Special frauds targeting the elderly are on the rise, and traditional prevention systems have struggled to detect fraud in real time or respond quickly. Furthermore, it is important to not only detect and warn of signs of fraud, but also to record the contents of calls so that they can be reviewed later. There is also a need for systems that allow users to easily manage blacklists and ensure data safety and privacy.

[1748] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1749] In this invention, the server includes: means for receiving voice data transmitted from the terminal in real time and converting the voice data into text data using a voice recognition engine; means for analyzing the converted text data using a natural language processing algorithm to detect suspicious patterns; means for sending visual and audio alerts to the user's terminal when a suspicious pattern is detected; means for sending notifications to the terminals of pre-registered family members or guardians when a suspicious pattern is detected; and means for generating a summary report of the call content after the call ends and storing it in a user database. This enables the detection and prevention of fraud against elderly people in real time, and allows the content of the call to be reviewed later, enabling more effective fraud prevention. It also allows users to easily manage their blacklist, ensuring data security and privacy.

[1750] "Terminal" means a device used by a user to access the special fraud prevention system.

[1751] A "server" is a central computer system that receives and processes data sent from terminals.

[1752] "Audio data" refers to audio information of the contents of a call captured from a terminal.

[1753] A "voice recognition engine" is a software or hardware technology that converts voice data into text data.

[1754] "Text data" is character string information converted from voice data by a voice recognition engine.

[1755] A "natural language processing algorithm" is a computer program that analyzes text data and detects specified patterns or intent.

[1756] A "suspicious pattern" is a combination of specific words, phrases, or context that indicates possible fraud.

[1757] "Visual and audio alerts" are screen displays and audio notifications that notify the user of the detection of a suspicious pattern.

[1758] "Family and guardians" are trusted contacts registered by the user who will receive notifications in the event of an emergency.

[1759] "Notifications" are warning messages sent to family members or guardians when suspicious patterns are detected.

[1760] A "summary report" is a document generated after a call ends that includes a summary of the call, any suspicious pattern detections, and a history of alerts and notifications.

[1761] A "blacklist" is a list where users register phone numbers that may be fraudulent.

[1762] "Database" means a structured data storage for managing information collected and stored by a system.

[1763] The present invention is a special fraud prevention system for elderly people. This system works with the cooperation of terminals, a server, and users.

[1764] 1. Initial device setup

[1765] The user installs a special fraud prevention application on their device. This application has the function of capturing audio data while the user is on a call and sending it to a server in real time. After installation, the user launches the application for the first time and registers the contact information of their family or guardian. They also perform initial setup to add phone numbers that may be fraudulent to a blacklist. This completes the basic preparations for fraud prevention.

[1766] 2. Processing at the start of a call

[1767] The device starts capturing the audio data of the call as soon as the user starts the call, then the captured audio data is serialized into protocol buffers (Protobuf) or JSON format and streamed to the server in real time.

[1768] 3. Real-time audio analysis on the server

[1769] The server receives voice data sent from the device in real time. The received voice data is converted into text data using a speech recognition engine such as Google Cloud Speech-to-Text. The text data is then analyzed using natural language processing (NLP) algorithms (e.g., spaCy or BERT). The NLP algorithms detect specific phrases and contextual patterns that may be indicative of fraud, and if suspicious patterns are detected, a warning flag is raised.

[1770] 4. Warnings and Notifications

[1771] If a warning flag is raised, the server sends visual and audio alerts to the device, allowing the user to recognize and be on guard against possible fraud. For example, a pop-up message or audio alert may appear on the device stating, "This call may be fraudulent. Please be careful." The server also sends warning notifications to the devices of family members or guardians who the user has registered in advance. This notification includes the message, "This user's call may be fraudulent. Please check immediately."

[1772] 5. Post-call processing

[1773] Once the call is completed, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent. The report is then stored in the user database for later review by the user or family.

[1774] 6. Blacklist Management

[1775] The application interface allows users to add, delete, and edit the blacklist phone numbers, allowing users to add newly received phone numbers that they believe to be fraudulent to the blacklist. The edited blacklist is immediately sent to the server and stored in the database.

[1776] 7. Data Security and Privacy

[1777] All data is protected using strong encryption algorithms such as AES-256 when transmitted between the server and the device, reducing the risk of personal information and call content being leaked to third parties. Users can also manage their call history and data usage settings within the app, ensuring privacy while using the system.

[1778] Examples of specific examples and prompts

[1779] Examples:

[1780] A special fraud prevention app installed on the device automatically sends the contents of the call to a server.

[1781] The server analyzes the call content in real time and detects suspicious patterns.

[1782] If detected, an alert will be sent to you and your family.

[1783] Generate and save detailed reports after the call ends.

[1784] Example prompt sentence:

[1785] "Please explain the voice analysis system to prevent seniors from falling victim to fraud."

[1786] "Please explain in detail how your system analyzes call content in real time to prevent fraud."

[1787] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1788] Step 1:

[1789] Install the application on your device.

[1790] The user installs a special fraud prevention application on the device. After installation is complete, the application is launched for the first time, and the user registers the contact information of family members or guardians and adds phone numbers that may be used for fraud to a blacklist. These settings are sent from the device to the server and saved in a database.

[1791] Input: Application installations, contact information, blacklist information

[1792] Output: Configuration data sent to the server

[1793] Step 2:

[1794] Capture the contents of the call.

[1795] When a user initiates a call, the device starts capturing the call audio in real time, and the captured audio data is serialized and sent to the server in streaming format.

[1796] Input: Call audio

[1797] Output: Serialized audio data

[1798] Step 3:

[1799] Convert the audio data into text data.

[1800] The server receives the voice data sent from the terminal and converts the voice data into text data using a voice recognition engine (e.g., Google Cloud Speech-to-Text).

[1801] Input: Serialized audio data

[1802] Output: Text data

[1803] Step 4:

[1804] Analyze text data.

[1805] The server then analyzes the converted text data using natural language processing (NLP) algorithms (e.g., spaCy or BERT) to detect specific phrases and contextual patterns and look for signs of fraud.

[1806] Input: Text data

[1807] Output: Detection result (whether or not there is a suspicious pattern)

[1808] Step 5:

[1809] Send a warning to the user.

[1810] If the server detects a suspicious pattern, it will send a visual and audio alert to the device. The user's device will display a pop-up message saying, "This is a potential fraudulent call. Please be careful," and will also play an audio alert.

[1811] Input: Detection result (whether or not there is a suspicious pattern)

[1812] Output: Visual and audio alerts to user terminal

[1813] Step 6:

[1814] Send notifications to family members and guardians.

[1815] If the server detects a suspicious pattern, it sends a notification to the device of a pre-registered family member or guardian saying, "The user's call may be fraudulent. Please check immediately."

[1816] Input: Detection result (whether or not there is a suspicious pattern)

[1817] Output: Notification to family and guardians

[1818] Step 7:

[1819] Generate a summary report of the call.

[1820] After the call is completed, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent.

[1821] Input: Call text data and detection results

[1822] Output: Summary report

[1823] Step 8:

[1824] Manage blacklists.

[1825] The application's interface allows users to add, remove, and edit potentially fraudulent phone numbers to the blacklist, which is then sent to the server and stored in a database.

[1826] Input: User edits blacklist information

[1827] Output: Updated blacklist information

[1828] Step 9:

[1829] Ensure data security and privacy.

[1830] All data sent between the server and the device is protected using strong encryption algorithms such as AES-256, reducing the risk of personal information and call content being leaked to third parties.

[1831] Input: All data sent and received

[1832] Output: Encrypted data

[1833] (Application example 1)

[1834] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1835] Elderly people and store employees are at high risk of fraud, especially phone fraud. Employees are required to detect suspicious patterns during calls and respond quickly, but current methods make it difficult to provide real-time warnings and notifications, preventing effective prevention. Therefore, there is a need for a system that can detect signs of fraud during voice calls in real time, immediately warn employees, and notify managers.

[1836] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1837] In this invention, the server includes means for receiving voice data transmitted from the terminal in real time and converting the voice data into text data using a voice recognition engine, means for analyzing the converted text data using a natural language processing algorithm to detect suspicious patterns, means for sending visual and audio alerts to the user's terminal when a suspicious pattern is detected, means for capturing voice data during calls received by employees of a physical store based on an application example of the technology and transmitting it to the server for analysis, means for sending a notification to an administrator terminal when a suspicious pattern is detected, and means for generating a summary report of the call content after the call is ended and storing it on the server, thereby enabling real-time detection of signs of fraud and emergency response.

[1838] "Terminal" refers to the communication equipment used by the user, which is a device that captures and transmits audio data.

[1839] "Voice Data" refers to digital information converted from sound wave signals captured during a call and transmitted in real time to a server.

[1840] "Real-time" means that processing occurs nearly simultaneously with the actual event, with minimal delay.

[1841] A "speech recognition engine" is a software or hardware system for converting voice data into text data.

[1842] "Text data" refers to character information converted by a voice recognition engine, and represents the content of voice data as a string of characters.

[1843] A "natural language processing algorithm" is a computer program that analyzes text data and understands its grammatical structure and meaning.

[1844] A "suspicious pattern" is a specific pattern of words or context that indicates possible fraud and is defined based on historical data.

[1845] A "visual alert" is a visual warning message that appears on a user's device to draw attention when a suspicious pattern is detected.

[1846] "Audio alert" is an audio warning signal intended to audibly alert the user.

[1847] "Administrator" refers to a person in charge of system operation and management in an organization such as a physical store.

[1848] A "call summary report" is a report generated after a call ends that includes the key points of the call and any suspicious pattern detections.

[1849] A "blacklist" is a list of potentially fraudulent phone numbers maintained by the system.

[1850] The present invention is a special fraud prevention system, and a specific embodiment will be described that allows employees in physical stores to quickly detect and respond to fraud risks while on the phone. This system functions through the cooperation of terminals, a server, and users (employees and managers). The operation of each element and their interactions will be described in detail below.

[1851] 1. Initial device setup

[1852] A special fraud prevention application is installed on the user's (employee's) device. This application has a function to transmit the audio of a call to a server in real time. When the user starts the application for the first time, they register the administrator's contact information and perform the initial settings for the blacklist.

[1853] 2. Processing at the start of a call

[1854] Audio capture and transmission

[1855] The device starts capturing the call as soon as it is initiated, and the audio data is then appropriately serialized and streamed to the server in real time.

[1856] 3. Real-time audio analysis on the server

[1857] Receiving voice data and recognizing voice

[1858] The server receives the voice data sent from the device, and converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1859] Natural language processing and suspicious pattern detection

[1860] The converted text data is then analyzed using natural language processing (NLP) algorithms (e.g., SpaCy, nltk). NLP algorithms detect specific phrases and contextual patterns that may be indicative of fraud. If a suspicious pattern is detected, a warning flag is raised.

[1861] 4. Warnings and Notifications

[1862] Alert the user

[1863] If a warning flag is raised, the server will send a visual and audio alert to the terminal, which will alert the user (employee) to the possibility of fraud and bring it to their attention.

[1864] Examples:

[1865] When a suspicious pattern is detected, a pop-up message will appear saying, "This is a potential fraudulent call. Please be careful," and an audio warning will also be issued.

[1866] Administrator Notification

[1867] In addition, the server sends a notification to the device of a pre-registered administrator, stating that the user's (employee's) call may contain suspicious content.

[1868] Examples:

[1869] A message is sent to the administrator's smartphone saying, "An employee's call may be fraudulent. Please check immediately."

[1870] 5. Post-call processing

[1871] Generate summaries of call data

[1872] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent.

[1873] Call data storage and management

[1874] The generated reports are stored in the user database so that users (employees) and administrators can review them later.

[1875] 6. Blacklist Management

[1876] Viewing and Editing the Blacklist

[1877] Users (employees) can use the app interface to view, add, delete, and edit the blacklisted phone numbers, which are then sent to the server and stored in a database.

[1878] Examples:

[1879] Users (employees) can add newly received phone numbers that they believe to be fraudulent to the blacklist, and the changes are immediately reflected on the server.

[1880] 7. Data Security and Privacy

[1881] Data encryption and protection

[1882] All data sent between the server and the device is protected using a powerful encryption algorithm, reducing the risk of personal information and call content being leaked to third parties.

[1883] User-controlled data

[1884] Users (employees) can manage their own call history and data usage settings within the app, allowing them to use the system while ensuring privacy.

[1885] Prompt Sentence Examples

[1886] "I want to develop a system that detects potential fraud for employees taking customer orders over the phone. This system would analyze the audio during the call in real time and alert employees and managers if it detects signs of fraud. Please write code using Python and PyAudio to capture audio data and send it to a server. The server would use a speech recognition and natural language processing engine. Please also include a notification function if a suspicious pattern is detected."

[1887] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1888] Step 1:

[1889] The user (employee) starts the special fraud prevention application on the terminal and performs the initial setup. This requires registering the administrator's contact information and initial setup of the blacklist. This inputs basic information into the terminal for the system to operate properly.

[1890] Input: Administrator contact information, blacklist information

[1891] Output: Device with initial setup completed

[1892] Step 2:

[1893] When a call is initiated, the device begins capturing audio data through the microphone, which is then serialized and streamed to the server in real time.

[1894] Input: Voice data during a call

[1895] Output: Real-time audio data sent to the server

[1896] Step 3:

[1897] The server receives the voice data sent from the device in real time and converts it into text data using a speech recognition engine (e.g., Google Cloud Speech-to-Text).

[1898] Input: Real-time audio data

[1899] Output: Text data converted by the speech recognition engine

[1900] Step 4:

[1901] The server then analyzes the converted text data using natural language processing (NLP) algorithms (e.g., SpaCy, nltk) to detect specific phrases and contextual patterns that may be indicative of fraud.

[1902] Input: Text data

[1903] Output: Suspicious pattern detection results

[1904] Step 5:

[1905] If a suspicious pattern is detected, the server sends visual and audio alerts to the user's (employee's) device, making them aware of possible fraud and alerting them to the situation.

[1906] Input: Suspicious pattern detection result

[1907] Output: Visual and audio alerts sent to the user's device

[1908] Step 6:

[1909] If the server detects a suspicious pattern, it also sends a notification to the device of a pre-registered administrator, stating that the call content may contain suspicious content.

[1910] Input: Suspicious pattern detection result

[1911] Output: Notification sent to administrator's terminal

[1912] Step 7:

[1913] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, and a history of any alerts or notifications sent.

[1914] Input: Text data from calls, suspicious pattern detection results, history of alerts and notifications sent

[1915] Output: Call summary report

[1916] Step 8:

[1917] The generated summary report is stored in the server's user database, where users (employees) and managers can review the report at a later time.

[1918] Input: Call Summary Report

[1919] Output: Saved summary report

[1920] Step 9:

[1921] Users (employees) can add new phone numbers to the blacklist, delete them, or edit them. The edited blacklist is sent from the terminal to the server and stored in the database.

[1922] Input: Blacklist edit information

[1923] Output: Updated blacklist

[1924] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1925] The present invention combines an emotion engine with a special fraud prevention system targeted at elderly people, and functions with the cooperation of a terminal, a server, and a user. Specific embodiments of the present invention will be described below.

[1926] 1. Initial device setup

[1927] A special fraud prevention application is installed on the user's device. This application has the function of transmitting the voice during a call to a server in real time and the function of recognizing the user's emotions using an emotion engine. When the user starts the application for the first time, they register the contact information of their family or guardian and perform the initial settings for the blacklist.

[1928] 2. Processing at the start of a call

[1929] Audio capture and transmission

[1930] The device starts capturing the call as soon as it is initiated, and the audio data is then appropriately serialized and streamed to the server in real time.

[1931] 3. Real-time voice analysis and emotion recognition on the server

[1932] Receiving voice data and recognizing voice

[1933] The server receives the voice data sent from the terminal and converts the received voice data into text data using a voice recognition engine.

[1934] Natural language processing and suspicious pattern detection

[1935] The converted text data is then analyzed using natural language processing (NLP) algorithms, which detect specific phrases and contextual patterns that may be indicative of fraud. If a suspicious pattern is detected, a warning flag is raised.

[1936] Emotion recognition by emotion engine

[1937] In parallel, the voice data is fed into an emotion engine to recognize the user's emotions. The emotion engine analyzes voice characteristics such as tone, pitch, and speed to identify the user's emotional state.

[1938] 4. Warnings and Notifications

[1939] Alert the user

[1940] When a warning flag is raised, the server sends visual and audio alerts to the device, whose content and format are adaptively changed based on the user's emotions as recognized by the emotion engine.

[1941] Examples:

[1942] If a suspicious pattern is detected and the user expresses anxiety or fear, a highlighted alert will be displayed stating, "Urgent attention required. Possible scam."

[1943] Notification to family and guardians

[1944] In addition, the server sends a notification to the device of a pre-registered family member or guardian, informing them that the user's call may contain suspicious content and including the user's emotional information as recognized by the emotion engine.

[1945] Examples:

[1946] A message will be sent to the family member's smartphone saying, "A user has made a potentially fraudulent call and we are very concerned. Please check immediately."

[1947] 5. Post-call processing

[1948] Generate summaries of call data

[1949] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, the perceived emotional state, and a history of any alerts or notifications sent.

[1950] Call data storage and management

[1951] The generated reports are stored in the user database so that users and their families can review them later.

[1952] 6. Blacklist Management

[1953] Viewing and Editing the Blacklist

[1954] Users can view, add, remove, or edit the blacklisted phone numbers through the app's interface, which is then sent to the server and stored in a database.

[1955] Examples:

[1956] Users can add newly received phone numbers they suspect to be fraudulent to a blacklist, and the changes are immediately reflected on the server.

[1957] 7. Data Security and Privacy

[1958] Data encryption and protection

[1959] All data sent between the server and the device is protected using a powerful encryption algorithm, reducing the risk of personal information and call content being leaked to third parties.

[1960] User-controlled data

[1961] Users can manage their own call history and data usage settings within the app, allowing them to use the system while ensuring their privacy.

[1962] As described above, this invention can significantly reduce the risk of seniors becoming victims of fraud by integrating real-time call content analysis, a warning system, and emotion recognition. It also provides a user-friendly interface for easy use.

[1963] The processing flow will be explained below.

[1964] Processing at the start of a call

[1965] Step 1: Capture the call audio (on your device)

[1966] 1. Detect the start of a call.

[1967] 2. The system will begin the process of capturing the audio during the call.

[1968] 3. Convert the captured audio data into an appropriate format (e.g., buffer).

[1969] Step 2: Sending audio data (terminal)

[1970] 1. Serialize the converted audio data.

[1971] 2. Stream audio data to the server in real time.

[1972] Real-time analysis of voice data and emotion recognition

[1973] Step 3: Receiving audio data (server)

[1974] 1. Receive the audio data sent from the device.

[1975] 2. Deserialize the received audio data into the appropriate format.

[1976] Step 4: Speech Recognition (Server)

[1977] 1. Pass the deserialized voice data to the speech recognition engine.

[1978] 2. Convert the audio data into text data.

[1979] Step 5: Analysis using natural language processing (NLP) algorithms (server)

[1980] 1. The converted text data is fed into the NLP algorithm.

[1981] 2. Analyze the text using NLP algorithms to detect suspicious patterns (indicators of fraud).

[1982] 3. Raise a warning flag if a suspicious pattern is detected.

[1983] Step 6: Emotion recognition by the emotion engine (server)

[1984] 1. Input the voice data into the emotion engine.

[1985] 2. The emotion engine analyzes the tone, pitch, and rate of speech to identify the user's emotional state.

[1986] 3. The emotional state is recognized and recorded in a database.

[1987] Handling notifications and alerts

[1988] Step 7: Sending alerts to users (server)

[1989] 1. If a warning flag is raised, the server sends a visual and audio alert to the user terminal.

[1990] 2. The content and format of the alerts are adaptively changed based on the user's emotions recognized by the emotion engine.

[1991] Examples:

[1992] If anxiety or fear is detected, a highlighted alert will appear saying, "Urgent attention needed. Possible scam."

[1993] Step 8: Sending notifications to family members and guardians (server)

[1994] 1. If a warning flag is raised, a notification will be sent to the device of a pre-registered family member or guardian.

[1995] 2. The notification will include information about potentially suspicious content in the user's call and the perceived emotional information.

[1996] Examples:

[1997] A message will be sent to family members' smartphones saying, "We are very concerned that a user has made a potentially fraudulent call. Please check immediately."

[1998] Post-call processing

[1999] Step 9: Generate summary of call data (server)

[2000] 1. Once the call is complete, generate a call summary report.

[2001] 2. The report should include a summary of the call, any suspicious patterns detected, any emotional states recognized, and a history of notifications sent.

[2002] Step 10: Storing call data (server)

[2003] 1. The generated call summary is saved in the user database.

[2004] 2. The saved data will be available for review by the user or family at a later date.

[2005] Blacklist Management

[2006] Step 11: View and Edit Blacklist (User)

[2007] 1. The user accesses the app's blacklist management screen.

[2008] 2. View the current blacklist.

[2009] 3. User can add new phone numbers, delete existing numbers and perform edit operations.

[2010] Step 12: Update the Blacklist (Server)

[2011] 1. When a user edits the blacklist, the updates are sent to the server.

[2012] 2. The server saves the new blacklist contents in the database.

[2013] 3. The updated blacklist will be used the next time you receive a call.

[2014] Data Security and Privacy

[2015] Step 13: Encrypt and Secure Your Data (Server)

[2016] 1. Protect data sent between the server and the device with a strong encryption algorithm.

[2017] 2. The encryption technology used prevents third parties from eavesdropping or tampering with the data.

[2018] Step 14: User-managed data (User)

[2019] 1. Users can manage their call history and data usage settings within the app.

[2020] 2. You can take actions to ensure privacy, such as deleting data and changing settings.

[2021] Example 2

[2022] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2023] Special frauds targeting the elderly have been increasing in recent years, making countermeasures urgently needed. The elderly are particularly susceptible to the sophisticated tactics of fraudsters, and the damage they cause affects their families and society as a whole. Furthermore, existing fraud prevention systems lack the accuracy to detect suspicious patterns and have difficulty responding in real time. Furthermore, they lack appropriate alert functions that take into account the user's emotional state, which can prevent the elderly from responding appropriately in emergencies.

[2024] The identification process by the identification processing unit 290 of the data processing device 12 in Example 2 is realized by the following means. In this invention, the server includes: means for receiving voice data transmitted from a terminal in real time and converting the voice data into text data using a voice recognition module; means for analyzing the converted text data using a natural language analysis algorithm to detect suspicious patterns; means for identifying the user's emotional state using an emotion analysis engine when a suspicious pattern is detected; means for sending adaptive visual and audio alerts to the terminal based on the user's emotional state; and means for sending notifications to pre-registered contacts when a suspicious pattern is detected. This significantly reduces the risk of elderly people becoming victims of special fraud and enables rapid response in real time. Furthermore, alerts that take emotional state into account allow users to take appropriate action according to the situation.

[2025] "Audio data" is data that represents an audio signal transmitted from a terminal in digital form.

[2026] A "voice recognition module" is software or hardware for converting voice data into text data.

[2027] "Text data" refers to data that includes character information converted by a voice recognition module.

[2028] A "natural language analysis algorithm" is a set of techniques or processes that analyze text data to detect specific words, phrases, and contexts.

[2029] A "suspicious pattern" is a pattern of specific words, context, or behavior that suggests fraud or wrongdoing.

[2030] An "emotion analysis engine" is a system that recognizes and identifies a user's emotional state from voice and text.

[2031] A "visual alert" is a visual display of a warning message or reminder that is displayed on a device's display.

[2032] A "voice alert" is a warning message or warning alert issued by the device.

[2033] "Registered Contacts" refers to information about family members, guardians, and other emergency contacts that the user has previously registered in the system.

[2034] "Real-time" means that processing and communication occur almost simultaneously with real time.

[2035] "Terminal" refers to a communication device or computer system used by a user, and is a device that constitutes part of this system.

[2036] "Server" refers to a central computer system that receives and processes data from terminals.

[2037] The present invention combines an emotion analysis function with a special fraud prevention system targeted at elderly people, and is a system that functions with the cooperation of terminals, servers, and users. Specific embodiments of the present invention are described in detail below.

[2038] A special fraud prevention application is installed on the user's device. This application has built-in functions for transmitting voice to a server in real time and for recognizing the user's emotions using an emotion analysis engine. When the application is first launched, the user registers the contact information of family members or guardians and sets up a blacklist.

[2039] 1. Initial device setup

[2040] Users install a special fraud prevention application on their device and enter the contact information of their family members or guardians and their blacklist within the application. Once registration is complete, this data is sent to a server and stored in a database.

[2041] 2. When the call starts

[2042] When a user initiates a call, the device automatically starts capturing the call. During the call, audio data is sent to the server in real time, where it is properly serialized and streamed using WebSockets.

[2043] 3. Real-time analysis by the server

[2044] The server receives the voice data sent from the device in real time and converts it into text data using the Google Speech-to-Text API. The converted text data is analyzed using a natural language analysis algorithm (e.g., SpaCy) to detect suspicious patterns. If a suspicious pattern is detected, a warning flag is raised. In parallel, the voice data is sent to an emotion analysis engine to identify the user's emotional state.

[2045] For example, if a user makes a call saying, "Please transfer money," the natural language analysis algorithm will detect potentially fraudulent keywords such as "money" and "transfer" and raise a warning flag.

[2046] 4. Sending alerts and notifications

[2047] When the server receives a warning flag, it sends a visual and audio alert to the user's device. The content of the alert changes adaptively based on the user's emotions, as recognized by an emotion analysis engine. For example, if the user expresses anxiety or fear, a highlighted alert will be displayed stating, "Urgent attention required. Possible fraud." The server also sends notifications to pre-registered contacts to inform the user of a suspicious call.

[2048] For example, a message may be sent to a family member's smartphone saying, "We are very concerned that a user has made a potentially fraudulent call. Please check immediately."

[2049] 5. Post-call processing

[2050] Once the call is over, the server generates a summary report of the call, including a summary of the call, any suspicious patterns detected, the perceived emotional state, and a history of any alerts or notifications sent. The report is stored in the user database for later review by the user or family members.

[2051] 6. Blacklist Management

[2052] Through the application interface, users can view, add, delete and edit the blacklist, and these changes are sent to the server and stored in a database.

[2053] For example, when a user adds a newly received phone number that they believe to be fraudulent to the blacklist, the changes are immediately reflected on the server.

[2054] 7. Data Security and Privacy

[2055] Data transmitted between the server and the device is protected using strong encryption algorithms such as AES-256. In addition, users can manage their call history and data usage settings within the app, ensuring their privacy and allowing them to use the system with peace of mind.

[2056] The system significantly reduces the risk of seniors becoming victims of fraud and enables real-time responses, with adaptive alerts based on emotional state allowing users to take appropriate action quickly.

[2057] Example prompt sentence:

[2058] "I'm thinking about a system to prevent special frauds against the elderly. Could you please give me an overview of the system that analyzes phone calls in real time and issues a warning if there are signs of fraud?"

[2059] The flow of the identification process in the second embodiment will be described with reference to FIG.

[2060] Step 1:

[2061] The user installs the special fraud prevention application on their device. When they first launch it, they enter the contact information and blacklist of their family members or guardians to register them in the application.

[2062] Specifically, the user downloads and installs the application. After launching the application, they enter the necessary contact and blacklist information on the registration screen that appears, and then press the "Save" button. This information is sent to the server and saved in a database.

[2063] Input: User contact information, blacklist information

[2064] Output: Preferences information stored in the server database

[2065] Step 2:

[2066] When a user initiates a call, the device automatically prepares to capture the call content.

[2067] Specifically, the app uses the mobile phone's API to detect when a new call starts, and the moment the call starts, the app automatically launches in the background.

[2068] Input: Call Start Event

[2069] Output: Ready to capture conversation

[2070] Step 3:

[2071] The device captures the audio data of the call and streams it in real time to the server, where it is appropriately serialized and transmitted using WebSockets.

[2072] Specifically, the device stores the call audio in a buffer. Whenever the data in this buffer reaches a certain amount, the audio data is streamed via WebSocket.

[2073] Input: Call audio data

[2074] Output: Serialized audio data sent to the server

[2075] Step 4:

[2076] The server receives the voice data transmitted from the terminal in real time and converts the voice data into text data using a voice recognition module.

[2077] Specifically, when the server receives the voice data, it calls the Google Speech-to-Text API to perform speech recognition, and the results are returned as text data.

[2078] Input: Serialized audio data

[2079] Output: Text data

[2080] Step 5:

[2081] The server then analyzes the converted text data using natural language analysis algorithms to detect suspicious patterns, while a sentiment analysis engine identifies the user's emotional state.

[2082] Specifically, the server analyzes the text data using SpaCy to detect suspicious patterns based on specific keywords and context, while simultaneously sending the audio data to an emotion analysis engine to determine the user's emotional state based on their tone and pitch.

[2083] Input: Text data, audio data

[2084] Output: Presence or absence of suspicious patterns, emotional state

[2085] Step 6:

[2086] If a suspicious pattern is detected, the server sends adaptive visual and audio alerts to the user's device based on their emotional state.

[2087] Specifically, the server checks the warning flag, and if the user shows signs of anxiety or fear, it sends an alert message to the user's device stating, "Urgent attention required. Possible fraud."

[2088] Input: Suspicious pattern detection results, emotional state

[2089] Output: An adaptive alert message sent to the user's device.

[2090] Step 7:

[2091] If a suspicious pattern is detected, the server will also send a notification to pre-registered contacts.

[2092] Specifically, the server will notify the user's family or guardian with a message stating, "The user has made a potentially fraudulent call and is very concerned. Please check immediately."

[2093] Input: Suspicious pattern detection results, emotional state

[2094] Output: Notification message sent to registered contacts

[2095] Step 8:

[2096] Once the call is over, the server generates a summary report of the call and stores it in the user database, including a summary of the call, any suspicious patterns detected, emotional state, and a history of any alerts or notifications sent.

[2097] Specifically, the server generates a summary report based on the full text of the call and emotion recognition data and stores it in a database.

[2098] Input: Full text data of calls, emotion recognition data

[2099] Output: Generated summary report

[2100] Step 9:

[2101] Users can view, add, delete and edit their blacklists, and these changes are sent to the server and stored in the database.

[2102] Specifically, when a user accesses the "Blacklist Management" menu in the app, enters a new phone number, and presses the "Add" button, the number is added to the blacklist. This information is sent to the server and stored in a database.

[2103] Input: User-modified blacklist

[2104] Output: Blacklist information updated in the server database

[2105] (Application example 2)

[2106] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[2107] Seniors need protection from fraud and suspicious transactions when shopping in brick-and-mortar stores. However, current systems lack real-time monitoring and emotion recognition in brick-and-mortar stores, putting seniors at high risk of fraud. Furthermore, existing prevention systems lack adaptive warnings that take into account the user's emotional state, preventing appropriate responses in emergencies.

[2108] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[2109] In this invention, the server includes means for receiving voice data transmitted from the terminal in real time and converting the voice data into text data using a voice recognition engine, means for analyzing the converted text data using a natural language processing algorithm to detect suspicious patterns, means for sending visual and audio alerts to the user's terminal when a suspicious pattern is detected, means for sending notifications to terminals of pre-registered family members or guardians when a suspicious pattern is detected, means for analyzing the voice data using an emotion recognition engine to identify the user's emotional state, and means for adaptively changing the corresponding alert format when the user expresses anxiety. This allows elderly people to enjoy shopping in physical stores without worrying about fraud or suspicious transactions.

[2110] A "terminal" is a device that captures audio and video data and transmits it to a server in real time.

[2111] "Voice data" refers to data that records voice information such as calls and conversations in digital format.

[2112] "Real-time" is a term that refers to data being processed immediately as it is acquired, without delay.

[2113] A "voice recognition engine" is a general term for software and algorithms that analyze voice data and convert its contents into text data.

[2114] "Text data" refers to character information converted from voice data by a voice recognition engine.

[2115] A "natural language processing algorithm" is a set of computational methods that allow a computer to process human language and analyze its meaning.

[2116] A "suspicious pattern" refers to a combination of specific words, phrases, or context that may indicate fraud or misconduct.

[2117] "Visual and audio alerts" means visual indications and audio notifications intended to alert the user.

[2118] An "emotion recognition engine" is a technology that uses audio and video data to analyze a user's emotional state.

[2119] "Adaptively changing" means automatically adjusting the system's output and behavior according to the situation or conditions.

[2120] "Family or guardian" refers to a person who supports the user and whose contact information has been registered in the system in advance.

[2121] "Alert Format" means a particular method or means of notifying a user of an urgent event.

[2122] The present invention combines an emotion recognition engine with a special fraud prevention system targeted at elderly people. Specific embodiments are described below.

[2123] 1. System Configuration

[2124] Hardware used

[2125] Smartphone

[2126] Smart Glasses

[2127] Software used

[2128] Speech recognition engine: Google Cloud Speech-to-Text API

[2129] Natural Language Processing Algorithm: BERT Model

[2130] Emotion Recognition Engine: Affectiva API

[2131] Communication protocol: WebRTC

[2132] Alerting system: Firebase Cloud Messaging (FCM)

[2133] 2. Specific implementation method of the system

[2134] User device initial settings

[2135] A fraud prevention application is installed on the user's smartphone or smart glasses. This application has the ability to capture audio and video data in real time and send it to a server. When the application is launched for the first time, the user registers the contact information of family members or guardians and performs initial settings for the blacklist.

[2136] Initiating calls and conversations

[2137] As soon as a call or in-store interaction is initiated, the device begins capturing audio and video data, which is serialized in real time and streamed to the server.

[2138] Data analysis on the server

[2139] The server receives the data sent from the device. The voice data is converted to text data using the Google Cloud Speech-to-Text API. The converted text data is analyzed using a natural language processing algorithm using the BERT model to detect suspicious patterns. If a suspicious pattern is detected, a warning flag is raised.

[2140] In parallel, the voice data is fed into an emotion recognition engine using the Affectiva API to analyze the user's emotional state. Based on the detailed analysis, if the user is expressing anxiety or fear, the alert format will be adaptively changed.

[2141] Alerts and Notifications

[2142] If a suspicious pattern is detected, the server will send a visual and audio alert to the device, such as an alert saying "Urgent attention required. Possible fraud."

[2143] Additionally, notifications will be sent to pre-registered family members and guardians on their devices, stating, "A user has made a potentially fraudulent call, and we are very concerned. Please check immediately."

[2144] Post-call processing

[2145] Once the call or interaction is over, the server generates a summary report of the call and stores it in a database for later review by the user or family members, including a summary of the call, any suspicious pattern detections, emotional state, and a history of any alerts or notifications sent.

[2146] Managing the Blacklist

[2147] Users can view, add, remove, and edit the blacklisted phone numbers and contacts through the app's interface, which is then sent to the server and stored in a database.

[2148] 3. Examples and prompts

[2149] Specific examples

[2150] If a scam indicator is detected in a physical store, such as "You need to give us your credit card details right away" or "You need to complete this process now to get a big discount!"

[2151] The user's smart glasses will play a message saying "Urgent attention required. Possible scam" along with an alert sound.

[2152] A notification will be sent to the family member's smartphone saying, "A user is engaged in a potentially fraudulent transaction and we are very concerned. Please check with the store immediately."

[2153] Prompt Sentence Examples

[2154] Input data: Audio data and real-time streaming

[2155] Emotion recognition API: Affectiva

[2156] Natural Language Processing API: Google Cloud Speech-to-Text, BERT model

[2157] process:

[2158] Converts voice data into text and analyzes it for signs of fraud

[2159] Analyze the user's emotional state

[2160] Triggers warnings when it detects suspicious patterns and user concerns

[2161] output:

[2162] Visual and audio alerts on the user's device

[2163] Notifications on family and guardian devices

[2164] As described above, the present invention provides a system that protects elderly people from fraud and suspicious transactions in physical stores, allowing them to enjoy shopping with peace of mind.By recognizing user emotions and providing adaptive warnings, the system can respond quickly and appropriately in emergencies.

[2165] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[2166] Step 1:

[2167] When a call or conversation in a physical store is initiated, the device captures audio and video data using its microphone and camera. This data is streamed to a server in real time using the WebRTC protocol. The input is audio and video data, and the output is serialized streaming data.

[2168] Step 2:

[2169] The server receives the voice data sent from the device in real time. The received voice data is input into the Google Cloud Speech-to-Text API, which converts the voice into text data. The input is voice data, and the output is voice transcription data (text data).

[2170] Step 3:

[2171] The server analyzes text data using a natural language processing algorithm based on the BERT model to detect suspicious patterns. The input is text data, and the output is a flag indicating whether a suspicious pattern exists. Specifically, the server performs grammatical analysis of the text data to detect suspicious words and phrases.

[2172] Step 4:

[2173] The server simultaneously analyzes the voice data with an emotion recognition engine using the Affectiva API to identify the user's emotional state. The input is the voice data, and the output is a classification of the user's emotional state (e.g., anxiety, fear, relief, etc.). Specific operations include analyzing the tone, pitch, and speed of the voice.

[2174] Step 5:

[2175] The server sends visual and audio alerts to the device if a suspicious pattern is detected and the user's emotional state indicates anxiety or fear. The input is a suspicious pattern flag and a classification of the emotional state, and the output is a visual and audio alert. Specific behaviors include adaptively changing the content and format of the alert.

[2176] Step 6:

[2177] If a suspicious pattern is detected, the server also sends a notification to the devices of pre-registered family members or guardians. The input is a suspicious pattern flag and the user's emotional state, and the output is a notification message to the family or guardian. Specifically, the notification is sent using Firebase Cloud Messaging (FCM).

[2178] Step 7:

[2179] Once the call or conversation is over, the server generates a summary report of the call and stores it in a database for later review by the user or family. The input is the text data of the call or conversation, the results of suspicious pattern detection, emotional state, and a history of sent alerts and notifications, and the output is a summary report. Specifically, this data is compiled into a single document and stored in the database.

[2180] Step 8:

[2181] Users can use the app's interface to view, add, delete, and edit blacklisted phone numbers and contacts. The input is the user's instructions, and the output is the updated blacklist information. Specifically, after accepting the user's operation, the app sends the updated blacklist data to the server and updates the database.

[2182] The above processing steps enable real-time audio and video data analysis, emotion recognition, suspicious pattern detection, warning systems, and notification systems to work together in an integrated manner, allowing elderly people to enjoy shopping in physical stores with peace of mind.

[2183] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[2184] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[2185] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[2186] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[2187] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[2188] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[2189] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[2190] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[2191] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[2192] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[2193] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[2194] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[2195] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[2196] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[2197] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[2198] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[2199] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[2200] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[2201] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[2202] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[2203] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[2204] The following is further disclosed regarding the above embodiment.

[2205] (Claim 1)

[2206] a means for receiving voice data transmitted from the terminal in real time and converting the voice data into text data using a voice recognition engine;

[2207] means for analyzing the converted text data using a natural language processing algorithm to detect suspicious patterns;

[2208] a means of sending visual and audio alerts to the user's device if a suspicious pattern is detected;

[2209] A method for sending notifications to pre-registered family members or guardians' devices when suspicious patterns are detected;

[2210] A system including:

[2211] (Claim 2)

[2212] 2. The system according to claim 1, further comprising means for capturing voice data when a call is initiated on the terminal side and streaming the captured voice data to the server in real time.

[2213] (Claim 3)

[2214] 10. The system of claim 1, further comprising means for providing an interface that allows a user to add, delete, and edit phone numbers on the blacklist, and for updating the blacklist transmitted to the server.

[2215] "Example 1"

[2216] (Claim 1)

[2217] a means for receiving voice data transmitted from the terminal in real time and converting the voice data into text data using a voice recognition engine;

[2218] means for analyzing the converted text data using a natural language processing algorithm to detect suspicious patterns;

[2219] a means of sending visual and audio alerts to the user's device if a suspicious pattern is detected;

[2220] A method for sending notifications to pre-registered family members or guardians' devices when suspicious patterns are detected;

[2221] means for generating a summary report of the call after the call is completed and storing the report in a user database; ...

Claims

1. a means for receiving voice data transmitted from the terminal in real time and converting the voice data into text data using a voice recognition engine; means for analyzing the converted text data using a natural language processing algorithm to detect suspicious patterns; a means of sending visual and audio alerts to the user's device if a suspicious pattern is detected; A method for sending notifications to pre-registered family members or guardians' devices when suspicious patterns are detected; A system including:

2. 2. The system according to claim 1, further comprising means for capturing voice data when a call is initiated on the terminal side and streaming the captured voice data to the server in real time.

3. 10. The system of claim 1, further comprising means for providing an interface that allows a user to add, delete, and edit phone numbers on the blacklist, and for updating the blacklist transmitted to the server.

Images